Proof-of-concept implemenation for supporting multiple pack repositories #126
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since pack can sit in the additional folder, it no longer make sense to get the path of a repack soley from the pack_id. Path to the repack file must be obtained by giving the pack_id itself. This ensures that the repack file sit on the same filessytem as the pack that is being repacked, and allow safe handling at the end using hard link.
zhubonan
changed the title
Codecov Report
@@ Coverage Diff @@
## develop #126 +/- ##
===========================================
- Coverage 99.52% 98.79% -0.74%
===========================================
Files 8 8
Lines 1675 1736 +61
===========================================
+ Hits 1667 1715 +48
- Misses 8 21 +13
Continue to review full report at Codecov.
|
|
close since it is superseded by #133 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR allows the packs to be stored in additional locations (pack repositories) other than the main object store folder.
The main reason for allowing this is allow packs to be moved to different file systems (slow but spacious ones) to free up spaces in the main filesystem (usually on a SSD), in case of growing demand of storage space.
Packing loose objects and writing directly to the packs will only take place on the main storage to avoid any performance impact.
One exception is when repacking - this will be done directly inside the pack repository (on the same filesytem as the pack file to be repacked). This means there is no longer a single path for temporaray pack that is used for repacking (previously pack_id=-1), as its path on the file system now depends on the id of the pack that is being repacked.
The user is resposible for moving the packs, ideally when the repository is fully "offline". Online operation is still possible, but requires a sequence of copy-rename-unlink operations.
Todos: