Ask questionsZSTD_insertBtAndGetAllMatches: Assertion `memcmp(match, ip, matchLength) == 0' failed

Describe the bug Hit an assertion (zstd: ../lib/compress/zstd_opt.c:651: ZSTD_insertBtAndGetAllMatches: Assertion `memcmp(match, ip, matchLength) == 0' failed.) while compressing a file.

This will probably be closed as "can't reproduce" as the file is 2363663601800 B (~ 2.1 TiB) and confidential.

To Reproduce Steps to reproduce the behavior:

  1. zfs send my private checkpoint, producing a 2363663601800 byte Zstream.
  2. pipe it into zstd -T24 -19
  3. Wait ~ 4 hours.

Expected behavior No assertion.

Desktop (please complete the following information):

  • OS: Ubuntu 21.04/AMD64 (fully up-to-date as of today)
  • *** zstd command line interface 64-bits v1.4.8, by Yann Collet ***
  • (MD5 sum of zstd: 0ebc538aa70c5dfd46ac06edc24f0287)
  • Host: Ryzen 3900XT w/16 GiB

Additional context My bandwidth for testing this is limited (and it's a multi-hour process), but if you point me to a specific version to try (I can give it a shot).


Answer questions terrelln

As far as I can tell there haven't been any relevant changes to zstd_opt.c since v1.4.8 so this is likely still a problem, but it is possible it is fixed in the latest release. Although it seems this bug is very rare, and hard to trigger, it is a serious problem because it could potentially cause corruption.

I agree that it will be hard to debug since we can't reproduce it. But, I will leave this issue open for now. Hopefully our fuzzers will find a much smaller reproducer, if the bug is still present, and we can close this issue.

Github User Rank List