profile
viewpoint
Martin Pool sourcefrog @google https://sourcefrog.net/ end qualified immunity

sourcefrog/conserve 84

[beta] robust portable backup tool in Rust

sourcefrog/natsort 69

natural sort order string comparison: "a1" < "a12"

sourcefrog/idris-array 1

primitive flat arrays containing Idris values

sourcefrog/projecteuler 1

my projecteuler answers

sourcefrog/pydkim 1

Python library that implements DKIM (DomainKeys Identified Mail) email signing and verification

sourcefrog/aoc2018 0

Answers to some of https://adventofcode.com/2018/.

sourcefrog/brotli2-rs 0

Brotli encoders/decoers for Rust

sourcefrog/cargo 0

The Rust package manager

sourcefrog/cargo-sweep 0

A cargo subcommand for cleaning up unused build files generated by Cargo

PR opened sourcefrog/conserve

Refactor backup code
+292 -253

0 comment

21 changed files

pr created time in 4 days

create barnchsourcefrog/conserve

branch : cleanup-backup

created branch time in 4 days

issue openedsourcefrog/conserve

parallelize `referenced_blocks`

referenced_blocks builds a set of all blocks referenced by all indexes. In 0.6.8 this is done on a single thread but it could fairly easily be parallelized:

  • [ ] Read multiple bands in parallel.
  • [ ] Read hunks of indexes in parallel.

This will significantly help performance of conserve gc and conserve delete.

Not strictly the same but related:

  • [ ] There's no need to stitch indexes when finding referenced blocks, because this amounts to reading the same hunk twice. (However, truncated indexes are probably fairly rare so this is not too high of a priority.)

https://github.com/sourcefrog/conserve/blob/9f405dc19a88cd4c038ed74213964b04b427bad8/src/archive.rs#L238-L261

created time in 4 days

push eventsourcefrog/conserve

Martin Pool

commit sha 6a8367eaef2493c95c0d1d3fbb8f14ed185c23e1

Add FileCombiner

view details

push time in 5 days

push eventsourcefrog/conserve

Martin Pool

commit sha aea784aa3ac4721cf88f1174bd88b97b88214b84

Tidy up

view details

push time in 5 days

push eventsourcefrog/conserve

Martin Pool

commit sha 31cf0158d4808c4509fcb8973541acafa80c2416

Add a test for splitting backups across hunks

view details

Martin Pool

commit sha cd0ca30fccc74ff829043b934892bd2cb56da851

Check names in many-file tree

view details

Martin Pool

commit sha e53a69d4d3bde3aa0a9822b9086ef649e9403f05

Move responsibility for breaking index hunks to backup The backup code needs to know about index hunk boundaries so that it can combine and parallelize storage of data within that hunk. And so the higher-level backup code now tells the index when it's done.

view details

push time in 5 days

push eventsourcefrog/conserve

Martin Pool

commit sha b2af71eae534ebbc97cb9f14af74efe9017cb688

Move StoreFiles to backup.rs

view details

Martin Pool

commit sha 08036fd946084004aed064ac89edaa90cc1eb07e

Remove StoreFiles All we need here is a reusable buffer.

view details

push time in 5 days

push eventsourcefrog/conserve

Martin Pool

commit sha 265079358c3c66267e8d14e4c341d8484dba30ca

Refactor backup towards processing group-at-a-time This changes the backup file-count stats to include files that later failed to be stored.

view details

push time in 6 days

push eventsourcefrog/conserve

Martin Pool

commit sha 9f405dc19a88cd4c038ed74213964b04b427bad8

cargo update

view details

push time in 6 days

PR opened sourcefrog/conserve

combine small files into composite blocks

Fixes #66

+292 -253

0 comment

21 changed files

pr created time in 7 days

issue closedsourcefrog/conserve

delete named versions

conserve delete -b b1234 /backup/home.c6
  • [x] allow specifying multiple bands
  • [x] gc after deletion
  • [x] but have an option to not gc

closed time in 7 days

sourcefrog

issue commentsourcefrog/conserve

delete named versions

Fixed in 2b06548adcdd3f12713f9c03ca0b419f20c5d2f6 etc, and released in 0.6.8.

sourcefrog

comment created time in 7 days

push eventsourcefrog/conserve

Martin Pool

commit sha 931da364bdcc0a02d465dc81d8854fb4ac811098

Bump version to 0.6.8-pre

view details

Martin Pool

commit sha 2b06548adcdd3f12713f9c03ca0b419f20c5d2f6

Add `conserve delete` command

view details

Martin Pool

commit sha 1259118be9e53738fc4d053bb5fa17bf5211b4b9

Rename 'blackbox' tests to 'cli' to be more specific

view details

Martin Pool

commit sha d0645e1d892cdf787f938b239efaa9a63afa522f

Accept Windows ioerror message

view details

Martin Pool

commit sha bb8ad51a6b5af1bed994281fb150752df667d15a

Release 0.6.8

view details

Martin Pool

commit sha c3a1f861f0a8b085e923861826ffcdc0ea0fa1c9

Bump version to 0.6.9-pre

view details

Martin Pool

commit sha 44af4e73cf7bc472f39a4e074634c395ba49f540

`delete` is now implemented

view details

Martin Pool

commit sha 684ea0385c2c649d2cc56c185734ed989d0468a6

Move backup to a non-associated function Give it its own loop for walking the tree, so that it can combine small files.

view details

Martin Pool

commit sha d7ec2191f93d76b396b671bf01e7bc6d1e3793c8

Combine subtree and exclusions to ReadTree:iter_filtered

view details

Martin Pool

commit sha fa734dc8305e0baaad138d8d6317ab8b23b9ed36

Apply exclusions during iterator not in the tree To iterate a local tree's subtree, start directly there and descend, rather than walking the whole thing and throwing data away. MergeTrees works on entry iterators not trees.

view details

Martin Pool

commit sha 80b18586277b15a99db011712efc0021c96efb2e

Support size --exclude and size --bytes

view details

Martin Pool

commit sha ae744a1f9f14b358e65471b5ca45e1dca269d02b

Clean up declaration of exclusions

view details

Martin Pool

commit sha 30c4d88b1296ad624b5186d9814f093648c42147

Use Option<GlobSet> for excludes

view details

Martin Pool

commit sha 0aca76d67b2ba3d312bdd012728f53058f982c5c

Don't need exclusions at the index level

view details

Martin Pool

commit sha b0ee766534d1598613a52f2fb05986c4ed146ff7

Simplify backup code No longer uses WriteTree trait

view details

Martin Pool

commit sha 787395c6730abee0a3576486849cae586205851a

Remove old comment

view details

Martin Pool

commit sha e075bc00cf24d2ffd34e6ca848f673bc44f8d4f5

Add todo

view details

push time in 7 days

push eventsourcefrog/conserve

Martin Pool

commit sha c3a1f861f0a8b085e923861826ffcdc0ea0fa1c9

Bump version to 0.6.9-pre

view details

Martin Pool

commit sha 44af4e73cf7bc472f39a4e074634c395ba49f540

`delete` is now implemented

view details

push time in 7 days

release sourcefrog/conserve

v0.6.8

released time in 7 days

created tagsourcefrog/conserve

tagv0.6.8

[beta] robust portable backup tool in Rust

created time in 7 days

push eventsourcefrog/conserve

Martin Pool

commit sha 2b06548adcdd3f12713f9c03ca0b419f20c5d2f6

Add `conserve delete` command

view details

Martin Pool

commit sha 1259118be9e53738fc4d053bb5fa17bf5211b4b9

Rename 'blackbox' tests to 'cli' to be more specific

view details

Martin Pool

commit sha d0645e1d892cdf787f938b239efaa9a63afa522f

Accept Windows ioerror message

view details

Martin Pool

commit sha bb8ad51a6b5af1bed994281fb150752df667d15a

Release 0.6.8

view details

push time in 7 days

push eventsourcefrog/conserve

Martin Pool

commit sha 931da364bdcc0a02d465dc81d8854fb4ac811098

Bump version to 0.6.8-pre

view details

push time in 7 days

push eventsourcefrog/conserve

Martin Pool

commit sha d0645e1d892cdf787f938b239efaa9a63afa522f

Accept Windows ioerror message

view details

push time in 7 days

create barnchsourcefrog/conserve

branch : delete

created branch time in 7 days

delete branch sourcefrog/conserve

delete branch : validate-parallel

delete time in 10 days

delete branch sourcefrog/conserve

delete branch : validate-blocks

delete time in 10 days

delete branch sourcefrog/conserve

delete branch : garbage

delete time in 10 days

delete branch sourcefrog/conserve

delete branch : combine-blocks

delete time in 10 days

push eventsourcefrog/conserve

Martin Pool

commit sha 0689802782cc68ad8f5c8a3dd787cc20077b690a

Use Option<GlobSet> for excludes

view details

Martin Pool

commit sha 3367169e6a5aaa7fc83290ce7224ea00747f3fab

Don't need exclusions at the index level

view details

Martin Pool

commit sha 4b205283c5bd1fa4a1064b0578287fbffaa8cfc9

Simplify backup code No longer uses WriteTree trait

view details

Martin Pool

commit sha 1274a2d4db0fe371353658409023d1be7aec5075

Remove old comment

view details

Martin Pool

commit sha 94c92a559884ee35245496ab4ef6bc0e0bbcf1f6

Add todo

view details

push time in 11 days

push eventsourcefrog/conserve

Martin Pool

commit sha 391107fa0936a83806a91ca38f895a4c80739dc8

Clean up declaration of exclusions

view details

push time in 11 days

push eventsourcefrog/conserve

Martin Pool

commit sha df43cca62caa6018c5932b92e1d70fdce0887c82

Support size --exclude and size --bytes

view details

push time in 11 days

push eventsourcefrog/conserve

Martin Pool

commit sha 5c3414c63384d9d1de92ce65ef070e1fe16274c6

Combine subtree and exclusions to ReadTree:iter_filtered

view details

Martin Pool

commit sha 1acda9f1e138c64443bae04846431ee478f16d49

Apply exclusions during iterator not in the tree To iterate a local tree's subtree, start directly there and descend, rather than walking the whole thing and throwing data away. MergeTrees works on entry iterators not trees.

view details

push time in 11 days

create barnchsourcefrog/conserve

branch : combine

created branch time in 12 days

startedmxrch/GHunt

started time in 17 days

release sourcefrog/conserve

v0.6.7

released time in 18 days

created tagsourcefrog/conserve

tagv0.6.7

[beta] robust portable backup tool in Rust

created time in 18 days

push eventsourcefrog/conserve

Martin Pool

commit sha fef93f8fa2a25a09f29ab043645caea3b222b5ae

cargo update

view details

Martin Pool

commit sha 2c0d79a62b8384960264c9c01d39fa1e325a7cbb

Release 0.6.7

view details

push time in 18 days

startedlunasorcery/useless-machine

started time in 21 days

push eventsourcefrog/rsyn

Martin Pool

commit sha 145574f055638317673b1974933bd23417de4392

Delete unnecessary conversion #clippy

view details

push time in a month

push eventsourcefrog/rsyn

Martin Pool

commit sha e53c2eb8901b4a4a66229d057e125758247cac27

cargo update

view details

Martin Pool

commit sha ae057c250b6e7e9a06ebd9ebe99e44499d5ab550

Update lint name

view details

push time in a month

push eventsourcefrog/conserve

orhun

commit sha 3f251bd135c4aa95a8b26b81a0a784f8b3faa887

Update README.md about installation from AUR

view details

Martin Pool

commit sha b2086a642612b3d48c9688be4abf11d76473df28

AUR is specifically for Arch Linux

view details

Martin Pool

commit sha 4613e0bfab2cb007db50b30f268303f4c117d027

Merge pull request #150 from orhun/main Update README.md about installation from AUR

view details

Martin Pool

commit sha 4dfb9057d81f5cab2f8cfa13b779567135fabe4d

Add `conserve gc` Fixes #146

view details

push time in a month

push eventsourcefrog/conserve

Martin Pool

commit sha 4903b88df2968c5aae7596d2c6ff50779a0d0cd0

Design doc for deletion guards

view details

Martin Pool

commit sha 8933d4d6bcf91b9a370e42407ecc8e188a05cc82

Add DeleteGuard

view details

Martin Pool

commit sha 960ba372d2d832c7f46d2081c61e9d8daf696efc

Add `conserve gc` * Add DeleteGuard to prevent block deletion while a backup is underway. * Add Transport::remove_file * Add BlockDir::delete_block and compressed_size

view details

Martin Pool

commit sha a005e351cf414499c939c76a1bcdf497ea30fe9e

Add a caution not to delete while a backup is underway

view details

Martin Pool

commit sha 58a75dbeab989b5bbeff47839e89c0f361dbcd67

Rename to gc_lock::GarbageCollectionLock

view details

Martin Pool

commit sha aa2b870cc78d87697b433ae8fb439e83f893da19

Notes on GC_LOCK

view details

Martin Pool

commit sha 94c19fcc473041b202966926f82754a7f5a71848

Mark archive locked during GC For safety, we need to prevent new backups starting while a gc operation is underway, otherwise they might reference blocks that will soon be removed. This adds a GC_LOCK file in the archive directory which is present only while GC is underway. Only one GC can run at a time. The lock is removed on drop, and can be explictly broken. This is not yet wired into backups or the CLI.

view details

Martin Pool

commit sha 250263b64a5ca3bb2d182975d64c84bd39d391c9

Wire up `gc --break-lock`

view details

Martin Pool

commit sha 6279dc904e6b518e4a00fe7ee75a93867ecd0c8a

Don't start backups while GC_LOCK is held

view details

Martin Pool

commit sha 368763c4444f511069890edb0dcf262454341516

Parallelize measuring and deleting gc blocks

view details

Martin Pool

commit sha 4dfb9057d81f5cab2f8cfa13b779567135fabe4d

Add `conserve gc` Fixes #146

view details

push time in a month

PR merged sourcefrog/conserve

Add `conserve gc` topic:deletion type:feature

Fixes #146

+488 -14

1 comment

18 changed files

sourcefrog

pr closed time in a month

issue closedsourcefrog/conserve

delete unreferenced blocks (garbage collection)

Unreferenced blocks can be left behind when a backup is interrupted, and this is also a building block for deleting versions (#43).

We should add

conserve gc [--dry-run] ARCHIVE

This is not intended to be run while a backup is being written, but it should be safe even if this accidentally done, for example if the backup is made by a cron job. To do this:

  • [x] gc should not start if there is an incomplete backup, which might still be being written, as the backup job might assume the blocks present when it started are still present.
  • [x] backups should not start if a gc job is underway, because it might delete blocks from under them. This seems to require some kind of gc lock file.

In general I'd like to avoid lock files but for gc it seems difficult to do without it.

gc lock files seem to have extra complications if the gc job is abruptly stopped (how do we know to remove the lock) or if two gc jobs run at the same time.

closed time in a month

sourcefrog

issue closedsourcefrog/conserve

delete unreferenced blocks (garbage collection)

Unreferenced blocks can be left behind when a backup is interrupted, and this is also a building block for deleting versions (#43).

We should add

conserve gc [--dry-run] ARCHIVE

This is not intended to be run while a backup is being written, but it should be safe even if this accidentally done, for example if the backup is made by a cron job. To do this:

  • [x] gc should not start if there is an incomplete backup, which might still be being written, as the backup job might assume the blocks present when it started are still present.
  • [x] backups should not start if a gc job is underway, because it might delete blocks from under them. This seems to require some kind of gc lock file.

In general I'd like to avoid lock files but for gc it seems difficult to do without it.

gc lock files seem to have extra complications if the gc job is abruptly stopped (how do we know to remove the lock) or if two gc jobs run at the same time.

closed time in a month

sourcefrog

push eventsourcefrog/conserve

Martin Pool

commit sha 368763c4444f511069890edb0dcf262454341516

Parallelize measuring and deleting gc blocks

view details

push time in a month

push eventsourcefrog/conserve

Martin Pool

commit sha 6279dc904e6b518e4a00fe7ee75a93867ecd0c8a

Don't start backups while GC_LOCK is held

view details

push time in a month

push eventsourcefrog/conserve

orhun

commit sha 3f251bd135c4aa95a8b26b81a0a784f8b3faa887

Update README.md about installation from AUR

view details

Martin Pool

commit sha b2086a642612b3d48c9688be4abf11d76473df28

AUR is specifically for Arch Linux

view details

Martin Pool

commit sha 4613e0bfab2cb007db50b30f268303f4c117d027

Merge pull request #150 from orhun/main Update README.md about installation from AUR

view details

push time in a month

PR merged sourcefrog/conserve

Update README.md about installation from AUR platform:unix type:doc

Hi, I'm a Rust developer who maintains AUR packages in his spare time. Recently I came across this project and I really liked its manifesto and implementation. Thus I created the following packages for the installation of conserve on Arch Linux and other distributions that support installing packages from AUR:

This PR basically adds a section to README.md about installation from AUR.

Take care, orhun.

+6 -0

1 comment

1 changed file

orhun

pr closed time in a month

push eventorhun/conserve

Martin Pool

commit sha b2086a642612b3d48c9688be4abf11d76473df28

AUR is specifically for Arch Linux

view details

push time in a month

pull request commentsourcefrog/conserve

Update README.md about installation from AUR

Neat, thanks!

I think I'll change the text to make it more clear this is only for Arch Linux users...

orhun

comment created time in a month

push eventsourcefrog/conserve

Martin Pool

commit sha 250263b64a5ca3bb2d182975d64c84bd39d391c9

Wire up `gc --break-lock`

view details

push time in a month

push eventsourcefrog/conserve

Martin Pool

commit sha 94c19fcc473041b202966926f82754a7f5a71848

Mark archive locked during GC For safety, we need to prevent new backups starting while a gc operation is underway, otherwise they might reference blocks that will soon be removed. This adds a GC_LOCK file in the archive directory which is present only while GC is underway. Only one GC can run at a time. The lock is removed on drop, and can be explictly broken. This is not yet wired into backups or the CLI.

view details

push time in a month

push eventsourcefrog/conserve

Martin Pool

commit sha 58a75dbeab989b5bbeff47839e89c0f361dbcd67

Rename to gc_lock::GarbageCollectionLock

view details

Martin Pool

commit sha aa2b870cc78d87697b433ae8fb439e83f893da19

Notes on GC_LOCK

view details

push time in a month

startedctrlpvim/ctrlp.vim

started time in a month

startedarraypad/each

started time in a month

startedcdr/code-server

started time in a month

push eventsourcefrog/conserve

Martin Pool

commit sha c8209833fec48a501489e7fe79602b2c1b94b0ae

Bump version to 0.6.7-pre

view details

push time in 2 months

PR opened sourcefrog/conserve

Add `conserve gc`

Fixes #146

+340 -14

0 comment

16 changed files

pr created time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 960ba372d2d832c7f46d2081c61e9d8daf696efc

Add `conserve gc` * Add DeleteGuard to prevent block deletion while a backup is underway. * Add Transport::remove_file * Add BlockDir::delete_block and compressed_size

view details

Martin Pool

commit sha a005e351cf414499c939c76a1bcdf497ea30fe9e

Add a caution not to delete while a backup is underway

view details

push time in 2 months

issue openedsourcefrog/conserve

delete leftover index temp files

If a backup is interrupted, temporary files might be left behind in the index directory. These can be deleted immediately before the band is finalized.

Possibly, conserve gc could also look for them in already-complete index directories. They might have been left behind by versions before the check in the first paragraph was added.

There probably can't ever be very much space used by these, so it's not a high priority.

created time in 2 months

issue openedsourcefrog/conserve

delete leftover temporary block files

Temporary files might be left behind in the block directory if a backup is interrupted. conserve gc (from #146) should delete them too. This of course requires checking that no backup is underway.

created time in 2 months

issue openedsourcefrog/conserve

delete unreferenced blocks (garbage collection)

Unreferenced blocks can be left behind when a backup is interrupted, and this is also a building block for deleting versions (#43).

We should add

conserve gc ARCHIVE

created time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 8933d4d6bcf91b9a370e42407ecc8e188a05cc82

Add DeleteGuard

view details

push time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 4903b88df2968c5aae7596d2c6ff50779a0d0cd0

Design doc for deletion guards

view details

push time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha ac3456e140d5bd4893c5b6627569ab9f214bd375

derive Default for CheckOrder

view details

Martin Pool

commit sha 02e21df52b5e027711e433059f9b612ff9dea04a

Add Archive::band_exists and band_is_closed Allow checking for bands without creating a new Band object.

view details

Martin Pool

commit sha fe1c39f5db24bd397cba12fe0695ca4ebc018cb1

Add IndexHunkIter, including advance_to_after Read groups of index entries, in the order and granularity they occur in the index. This is a foundation for stitching indexes, and I think also gives a foundation for naturally parallelizing retrieval of files from the archive.

view details

Martin Pool

commit sha fe592777e31fda0440ed0b8e652adbf77c896949

Stitch incomplete indexes

view details

Martin Pool

commit sha 0d4fc6f6c75d194e42bdee973133f08daee3464c

StoredTree uses stitched indexes Any access to a stored tree will now stitch the indexes if necessary.

view details

Martin Pool

commit sha 34aed897bb0cbc185362d48f34773037eee42e4e

Validation is implemented Also reformat markdown

view details

Martin Pool

commit sha 2b874f91ec59a3f5743982c4f0edb7ca99fb132d

Mention that stitched indexes exist

view details

Martin Pool

commit sha acd753fd5b8c06383eda9665b3b4c0f449b8a65b

The --incomplete option is obsolete

view details

Martin Pool

commit sha 0e6261b572f89b95ad6bae1377caa804ce2bb888

Test stitching indexes split into multiple hunks

view details

Martin Pool

commit sha 45bc1378bf437d24dbe1a932be69f0cdd8a67ee0

Better tests for index stitching

view details

Martin Pool

commit sha 5e97684a4957659b2d73fbf3003bdcb72a368df9

Stitch incomplete indexes with predecessors Incomplete trees no longer need special handling in the UI. Instead, Conserve will show or restore the best-available version of the tree at that point in time. Fixes #80

view details

Martin Pool

commit sha 1ad352c4cc2cd0762d9839940fbd8a92be5f011f

Fix broken doctest

view details

Martin Pool

commit sha 8f62bfcb8150b43167c42f659e9f8f5e0abe5845

Reenable doctests

view details

Martin Pool

commit sha 78c05efe97e3baf4a0e5a60bbc9959698159cde6

cargo update

view details

Martin Pool

commit sha 5244e07e693111406c0adb351e08e56dbd5582bd

Prepare 0.6.5 release

view details

Martin Pool

commit sha 4336ff7ac383c0eb38ea265f691537ea693a927e

Bump version to 0.6.6-pre

view details

Martin Pool

commit sha d100d192160b262fbeca7d8ee7a2b6818abd5f9e

Start adding a strongly-typed BlockHash2

view details

Martin Pool

commit sha 21c3da1bacbee4e92f9f011a80c86acfdeb3b0c9

Silence clippy new_without_default for CheckOrder

view details

Martin Pool

commit sha 037f51436fcb52b01022220a011a307dd616c69c

impl Hash for BlockHash2

view details

Martin Pool

commit sha 36e7a3e3c2459262d5017b1abdc9ebdcf342fbba

impl From<Blake2bResult> for BlockHash2

view details

push time in 2 months

release sourcefrog/conserve

v0.6.6

released time in 2 months

created tagsourcefrog/conserve

tagv0.6.6

[beta] robust portable backup tool in Rust

created time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha faf8326bd44e0aedd2fe881d3681840bdee307a1

cargo update

view details

Martin Pool

commit sha 802840c69ebf198d44185e9a19b428feb1f65c5a

Prepare 0.6.6

view details

push time in 2 months

startedgraydon/rust-prehistory

started time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha d27437be2c06cf155ea61566ec554c4ec942fd22

Validate indexes in parallel Don't stop early if there are problems in the index. Just count the problems. Fixes #144

view details

push time in 2 months

issue closedsourcefrog/conserve

validate should check indexes in parallel

Validate currently checks indexes purely sequentially, but it would be faster to parallelize:

  1. check multiple indexes in parallel
  2. check multiple hunks from inside each index in parallel

In principle we could also parallelize checking different entries from inside a hunk, but this is probably not useful. Most of the time is spent getting the hunk from disk, decompressing, and deserializing it, and the per-entry work is very small.

Probably this is better to do after #141, the fix for which involves changing index validation to return a set of referenced addresses.


Checking multiple hunks inside an index in parallel is possible, but a little more complicated because we also want to check ordering constraints between hunks, so I deferred that to #145.

closed time in 2 months

sourcefrog

issue openedsourcefrog/conserve

validate could check hunks of an index in parallel

Following on from #144, validate could check multiple hunks from an index in parallel. This would help performance when there are fewer than NCPUS versions, but each very large - for example after the first backup of a large tree.

The hunks can basically all be checked independently, except that we do also want to check the last apath in one hunk sorts before the first apath in the successor.

created time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 939274e08befbc9ca177b35fe322e9873e39e6b0

Remove unnecessary error about empty index hunks Fixes #143

view details

Martin Pool

commit sha d27437be2c06cf155ea61566ec554c4ec942fd22

Validate indexes in parallel Don't stop early if there are problems in the index. Just count the problems. Fixes #144

view details

push time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 939274e08befbc9ca177b35fe322e9873e39e6b0

Remove unnecessary error about empty index hunks Fixes #143

view details

push time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha cc4a92f163a356de476d4de49593e2a0ba5384ff

Validate indexes in parallel Don't stop early if there are problems in the index. Just count the problems. Fixes #144

view details

push time in 2 months

create barnchsourcefrog/conserve

branch : validate-parallel

created branch time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 68fcb965a642034a010f80b53999c7b6c319ed1c

Remove unnecessary error about empty index hunks Fixes #143

view details

push time in 2 months

issue closedsourcefrog/conserve

"Index hunk is empty" shouldn't be reported as an error

Conserve v0.6.5-26-g9b82409 can show messages like this:

 conserve error: Index hunk "00000/000000319" is empty

This is excessively alarming: this was an encoding quirk from earlier versions but completely harmless. It probably shouldn't even be a warning.

closed time in 2 months

sourcefrog

startedRustCrypto/AEADs

started time in 2 months

issue openedsourcefrog/conserve

validate should check indexes in parallel

Validate currently checks indexes purely sequentially, but it would be faster to parallelize:

  1. check multiple indexes in parallel
  2. check multiple hunks from inside each index in parallel

In principle we could also parallelize checking different entries from inside a hunk, but this is probably not useful. Most of the time is spent getting the hunk from disk, decompressing, and deserializing it, and the per-entry work is very small.

Probably this is better to do after #141, the fix for which involves changing index validation to return a set of referenced addresses.

created time in 2 months

issue openedsourcefrog/conserve

"Index hunk is empty" should be a warning, not an error

Conserve v0.6.5-26-g9b82409 can show messages like this:

 conserve error: Index hunk "00000/000000319" is empty

This is excessively alarming: this was an encoding quirk from earlier versions but completely harmless. It probably shouldn't even be a warning.

created time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 9b82409f04a7f262958b83aae80f5a91cb0440f5

Delete old comment

view details

push time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 505c29dfe77574abc44f33dafe9b1122d076e5c0

Show progress percentage from work fraction

view details

Martin Pool

commit sha 6e512f1c345c495943635caaf34709f97967028e

Show estimated time remaining

view details

Martin Pool

commit sha b770057223f7bd3c32fee4f6ae25e2f92de84d0e

news for progress changes

view details

push time in 2 months

issue openedsourcefrog/conserve

Use stitched index as a basis

#80 didn't include using the stitched index as a basis for new backups, but we should, so that the backup after an interrupted backup can be faster.

created time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha aa98de06a63f68dd0ea163afd28edd4dd82f21d1

carge update

view details

push time in 2 months

issue openedsourcefrog/conserve

validate can complain about missing blocks, if a backup is made while validation is underway

This is harmless but might be needlessly alarming.

The particular case could be fixed by reversing the order: first see what blocks are referenced, and then check they're all present. If an index is added while this is underway, it won't be a problem.

Deletion of an index while validation is underway could still raise problems, but they're the same kind of problems if you deleted the index while it was being restored...

created time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 45da3b042e264b13cca944233a9fb2e37184694b

LocalTransport: read files in a single IO

view details

push time in 2 months

delete branch sourcefrog/conserve

delete branch : hashtype

delete time in 2 months

delete branch sourcefrog/conserve

delete branch : codecov

delete time in 2 months

delete branch sourcefrog/conserve

delete branch : update-actions

delete time in 2 months

delete branch sourcefrog/conserve

delete branch : travis

delete time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 38d238fb78f86bd03ff6b927af7aaf17302a2c8d

Use actions/checkout@v2

view details

push time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha d86758c3458c4af992510ea6d07e67d3f63683d1

Coverage changes are only informational

view details

push time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 27690395e99e638899a405ad7eaea6814eabe77c

Remove Travis

view details

Martin Pool

commit sha d86758c3458c4af992510ea6d07e67d3f63683d1

Coverage changes are only informational

view details

push time in 2 months

create barnchsourcefrog/conserve

branch : update-actions

created branch time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 27690395e99e638899a405ad7eaea6814eabe77c

Remove Travis

view details

push time in 2 months

create barnchsourcefrog/conserve

branch : codecov

created branch time in 2 months

create barnchsourcefrog/conserve

branch : travis

created branch time in 2 months

delete branch sourcefrog/pueue

delete branch : typo

delete time in 2 months

startedheadcrab-rs/headcrab

started time in 2 months

PR opened Nukesor/pueue

Fix typo
+1 -1

0 comment

1 changed file

pr created time in 2 months

create barnchsourcefrog/pueue

branch : typo

created branch time in 2 months

push eventsourcefrog/conserve

Martin Pool

commit sha 95295e93df1c7aab41b5d3e59feec87a1d2b3e43

Better progress during validate * Increment progress bar as blocks are checked * Change 'work' counts in progress bar to be usize * Don't need to show all filenames while checking indexes

view details

Martin Pool

commit sha 2523b7eeac43ac78d87546a6de8e5a16e5c29fdc

Add ProgressBar::new

view details

Martin Pool

commit sha 9dbfd9b03fcabda6ddf5206627031f7f7b01caa6

Move progress rate limiting into ProgressBar This avoids taking an extra mutex on the UI just to check if we can draw yet, and generally seems cleaner.

view details

push time in 2 months

more