Ask questionsDe-duplicate downloaded images (mostly for fiction.live)
fiction.live stories often have duplicate images in the downloaded epub, as every uploaded image will always receive a new URL, preventing simple deduping. If an author repeatedly reuses images this can significantly bloat the ebook.
Although they don't have the same URL, identical images will be byte-for-byte identical even after FanFicFare/Calibre image processing (resizing, grayscaling). Duplicates can therefore be detected by use of a fast cryptographic hash and only one copy kept in the ebook, all other sources rewritten to point to it.
This isn't by any means critical to supporting F.L, but it's nice to have smaller ebooks.
(I have an only-partly-functional F.L downloader written in Rust and I do this to keep sizes down; I use the BLAKE2 hash since it is fast and its length is configurable. Thanks for FanFicFare, it's the only way I read fanfics now.)
Answer questions AlyoshaVasilieva
Thanks a ton! Appears to be working fine so far and now the F.L ebooks are smaller.