profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/terrelln/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Nick Terrell terrelln @facebook zstd developer

terrelln/automata 4

Simulating and visualizing finite automata

terrelln/linux 4

Linux kernel source tree

pviolette3/jamstacks 0

full stack system for jamming stacks (www.jamstacks.club)

terrelln/abseil-cpp 0

Abseil Common Libraries (C++)

terrelln/air-types 0

Air Typing

terrelln/bril 0

an educational compiler intermediate representation

terrelln/btrfs-progs 0

Mirror of official repository of userspace BTRFS tools, plus development branches.

terrelln/buildroot 0

Buildroot, making embedded Linux easy. Note that this is not the official repository, but only a mirror. The official Git repository is at http://git.buildroot.net/buildroot/. Do not open issues or file pull requests here.

terrelln/cfg-expander 0

Exands CFG grammars

terrelln/clang 0

Mirror of official clang git repository located at http://llvm.org/git/clang. Updated every five minutes.

push eventterrelln/zstd-bench

Nick Terrell

commit sha 77b85a463f48b2f893df65e8ac1bada0bb4a9286

Fix comparison to compare against baseline

view details

push time in a day

push eventterrelln/zstd-bench

Nick Terrell

commit sha 61619c673ba9dafd8e07c0d91d46bf25213422a7

Fix typo in multi-comparison

view details

push time in a day

pull request commentfacebook/zstd

Fix DDSS Load

Maybe try force-pushing the branch, and see if that fixes the tests?

felixhandte

comment created time in a day

push eventterrelln/zstd-bench

Nick Terrell

commit sha f8aad00566382cb3d11d147413bc9f8f21844056

Add a diff view with --print-diff | -d

view details

push time in a day

push eventterrelln/zstd-bench

Nick Terrell

commit sha 595ee1246f77d47b5f769e9a3351edfca5cb2d2b

Remove dependency on nightly rust

view details

push time in 2 days

push eventterrelln/zstd-bench

Nick Terrell

commit sha b9928f24290019606c67352136b77116d9881a8f

Clarify the printing in the readme

view details

push time in 3 days

push eventterrelln/zstd-bench

Nick Terrell

commit sha 4b2f1c216e2ff7906b4f9f6280e75bafd8d8b0a3

Clarify the example

view details

push time in 3 days

pull request commentfacebook/zstd

Fix DDSS Load

LGTM once the tests pass!

felixhandte

comment created time in 3 days

issue commentfacebook/zstd

ZSTD_insertBtAndGetAllMatches: Assertion `memcmp(match, ip, matchLength) == 0' failed

As far as I can tell there haven't been any relevant changes to zstd_opt.c since v1.4.8 so this is likely still a problem, but it is possible it is fixed in the latest release. Although it seems this bug is very rare, and hard to trigger, it is a serious problem because it could potentially cause corruption.

I agree that it will be hard to debug since we can't reproduce it. But, I will leave this issue open for now. Hopefully our fuzzers will find a much smaller reproducer, if the bug is still present, and we can close this issue.

tommythorn

comment created time in 3 days

push eventterrelln/zstd-bench

Nick Terrell

commit sha 94c8c74266b10470eba23594a7d11bede9354dbd

Add examples and README

view details

push time in 3 days

create barnchterrelln/zstd-bench

branch : main

created branch time in 3 days

created repositoryterrelln/zstd-bench

Zstd benchmarking utilities

created time in 3 days

push eventfacebook/zstd

Nick Terrell

commit sha ba044bd6f12b22b3339731d6dc5a12476a24d1cc

[bug-fix] Fix a determinism bug with the DUBT The DUBT can be non-deterministic if an index is equal to `ZSTD_DUBT_UNSORTED_MARK`. Ensure that never happens by starting the indices at 2. This bug was found by the OSS-Fuzz determinism fuzzer. With this change the fuzzer test passes. And I've confirmed that this is the root cause, not just hiding the problem. Aside: This took me a long time to figure out, because I thought I had tried this first thing. But, apparantly I messed it up, because when I was going through it again with @felixhandte, I was pointing out that it wasn't the case, but it turns out it was. Credit to: OSS-Fuzz

view details

Nick Terrell

commit sha d2b5e5496d21570f565ad9cb3ea01112eb1bad88

Merge pull request #2726 from terrelln/oss-fuzz-fix [bug-fix] Fix a determinism bug with the DUBT

view details

push time in 3 days

PR merged facebook/zstd

[bug-fix] Fix a determinism bug with the DUBT CLA Signed

The DUBT can be non-deterministic if an index is equal to ZSTD_DUBT_UNSORTED_MARK. Ensure that never happens by starting the indices at 2.

This bug was found by the OSS-Fuzz determinism fuzzer. With this change the fuzzer test passes. And I've confirmed that this is the root cause, not just hiding the problem.

Aside: This took me a long time to figure out, because I thought I had tried this first thing. But, apparantly I messed it up, because when I was going through it again with @felixhandte, I was pointing out that it wasn't the case, but it turns out it was.

Credit to: OSS-Fuzz

+11 -8

0 comment

1 changed file

terrelln

pr closed time in 3 days

issue commentfacebook/zstd

Worse performance on aarch64 when building latest from source, vs older versions from yum repos

Thanks for the logs! I really wouldn't expect such a large regression. In fact, I would expect a speed improvement...

My suggestion would be to try a newer gcc version, gcc-4.8.5 is quite old. I'm not sure if a newer gcc will perform better or not. But, we can only "support" so many compilers for high performance. If a new version of gcc (gcc-11 is currently the latest, but at least gcc-8+) performs better, then the answer will just be to upgrade your compiler. However, if a new gcc still performs badly, then we will certainly look into it.

We do most of our benchmarking on x86-64. But, I know that @senhuang42 was benchmarking aarch64 performance for zstd-1.5.0 and saw performance improvements, not regressions. So if a compiler upgrade doesn't fix it, it will be worth an investigation.

mkjois

comment created time in 5 days

issue commentfacebook/zstd

Worse performance on aarch64 when building latest from source, vs older versions from yum repos

Hi @mkjois, can you print the output of zstd benchmark on the data you're compressing? All three of these zstd binaries will support the benchmark tool:

# Benchmark level 5
zstd -b5 -e5 /path/to/data

If you can't easily grab your real data, you can try the silesia corpus.

mkjois

comment created time in 6 days

push eventterrelln/zstd

Nick Terrell

commit sha ba044bd6f12b22b3339731d6dc5a12476a24d1cc

[bug-fix] Fix a determinism bug with the DUBT The DUBT can be non-deterministic if an index is equal to `ZSTD_DUBT_UNSORTED_MARK`. Ensure that never happens by starting the indices at 2. This bug was found by the OSS-Fuzz determinism fuzzer. With this change the fuzzer test passes. And I've confirmed that this is the root cause, not just hiding the problem. Aside: This took me a long time to figure out, because I thought I had tried this first thing. But, apparantly I messed it up, because when I was going through it again with @felixhandte, I was pointing out that it wasn't the case, but it turns out it was. Credit to: OSS-Fuzz

view details

push time in 9 days

Pull request review commentfacebook/zstd

[bug-fix] Fix a determinism bug with the DUBT

 ZSTD_checkDictValidity(const ZSTD_window_t* window,  MEM_STATIC void ZSTD_window_init(ZSTD_window_t* window) {     ZSTD_memset(window, 0, sizeof(*window));-    window->base = (BYTE const*)"";-    window->dictBase = (BYTE const*)"";-    window->dictLimit = 1;    /* start from 1, so that 1st position is valid */-    window->lowLimit = 1;     /* it ensures first and later CCtx usages compress the same */-    window->nextSrc = window->base + 1;   /* see issue #1241 */+    window->base = (BYTE const*)" ";

Yeah, CI compiler warnings were complaining about nextSrc = window->base + 1 being beyond the end of the index.

terrelln

comment created time in 9 days

PullRequestReviewEvent

push eventterrelln/zstd

Nick Terrell

commit sha 6994a66a8cbe8cd0201a5fe58766f771114a32b3

[bug-fix] Fix a determinism bug with the DUBT The DUBT can be non-deterministic if an index is equal to `ZSTD_DUBT_UNSORTED_MARK`. Ensure that never happens by starting the indices at 2. This bug was found by the OSS-Fuzz determinism fuzzer. With this change the fuzzer test passes. And I've confirmed that this is the root cause, not just hiding the problem. Aside: This took me a long time to figure out, because I thought I had tried this first thing. But, apparantly I messed it up, because when I was going through it again with @felixhandte, I was pointing out that it wasn't the case, but it turns out it was. Credit to: OSS-Fuzz

view details

push time in 9 days

PR opened facebook/zstd

[bug-fix] Fix a determinism bug with the DUBT

The DUBT can be non-deterministic if an index is equal to ZSTD_DUBT_UNSORTED_MARK. Ensure that never happens by starting the indices at 2.

This bug was found by the OSS-Fuzz determinism fuzzer. With this change the fuzzer test passes. And I've confirmed that this is the root cause, not just hiding the problem.

Aside: This took me a long time to figure out, because I thought I had tried this first thing. But, apparantly I messed it up, because when I was going through it again with @felixhandte, I was pointing out that it wasn't the case, but it turns out it was.

Credit to: OSS-Fuzz

+4 -3

0 comment

1 changed file

pr created time in 9 days

create barnchterrelln/zstd

branch : oss-fuzz-fix

created branch time in 9 days

pull request commentfacebook/zstd

Optimize zstd decompression by another x%

Hi @animalize, thanks for the benchmark!

We'd accept patches that improve Visual Studios performance without hurting clang/gcc performance or code legibility. However, VS is not a primary target, and we do not actively measure / optimize VS performance.

We already have more than enough trouble trying to keep clang/gcc performance high. Adding another compiler to that list makes the problem much harder. If anything, we'd prefer to drop a "supported performance" compiler rather than add one.

However, we do fully support VS for correctness.

danlark1

comment created time in 17 days

PR opened facebook/zstd

[RFC] Assembly implementation of 4X1 & 4X2 Huffman

Assembly in huf_decompress_x86_bmi2.S.

  • proba32.dat is 1MB of random symbols in [0, 32). Zstd compresses as all literals and uses 4X2 to decompress.
  • proba128.dat is 1MB of random symbols in [1, 128). Zstd compresses as all literals and uses 4X1 to decompress.
  • All files are compressed with level 1 to get the most # of literals.
File gcc-11 gcc-11 asm gcc-11 speedup clang-12 clang-12 asm clang-12 speedup
proba32.dat 1740 2123 +22% 1454 2136 +46%
proba128.dat 984 1312 +33% 962 1313 +36%
dickens 1063 1113 +4.7% 1031 1134 +9.9%
mozilla 999 1049 +5.0% 987 1061 +7.5%
mr 1260 1333 +5.8% 1195 1353 +13.3%
nci 1785 1793 +0.4% 1768 1793 +1.4%
ooffice 875 954 +9.0% 810 968 +19.5%
osdb 1213 1307 +7.7% 1215 1303 +7.2%
reymont 1076 1093 +1.5% 1064 1108 +4.1%
samba 1437 1465 +1.9% 1417 1481 +4.5%
sao 861 1010 +17.3% 867 1017 +17.3%
webster 1114 1146 +2.8% 1087 1164 +7%
xml 1793 1812 +1.0% 1751 1797 +2.6%
x-ray 1039 1256 +20% 877 1277 +45%

TODO: This is very much a demo / RFC. There is quite a bit more work to do before it is ready to merge.

  • Code cleanup
  • Lots of testing / fuzzing
  • Speed improvements - I haven't spent too much time on the 4X2 variant, it can probably be improved. I haven't found a better way to do 4X1, but I'm not an ASM expert, so maybe I'm missing something.
  • Better integration into the Makefiles - assembly compilation is currently hacked in
  • Integration into other build systems
+1082 -20

0 comment

5 changed files

pr created time in 18 days

push eventterrelln/zstd

Nick Terrell

commit sha 3c3a166e5cf583be2ef9568f43a0429731b24f63

1200 MB/s proba128.dat asm impl

view details

push time in 18 days

create barnchterrelln/zstd

branch : huf-asm

created branch time in 18 days

push eventfacebook/zstd

heitbaum

commit sha 7e429f62ba8643960d58589568d9b4ea1144ca03

[trace] remove zstd_trace.c reference from freestanding zstd_trace.c was removed as part of PR #2589

view details

Nick Terrell

commit sha 7b43e69b459e1c694caa0c49edbf028d3e5a9d8a

Merge pull request #2655 from heitbaum/patch-1 [trace] remove zstd_trace.c reference from freestanding

view details

push time in 25 days

PR merged facebook/zstd

[trace] remove zstd_trace.c reference from freestanding CLA Signed

zstd_trace.c was removed as part of PR #2589

+0 -1

3 comments

1 changed file

heitbaum

pr closed time in 25 days

pull request commentfacebook/zstd

[trace] remove zstd_trace.c reference from freestanding

Thanks for the PR @heitbaum!

heitbaum

comment created time in 25 days

push eventfacebook/zstd

Danila Kutenin

commit sha e855b78be6a9e2dff5fd81183c244835dc42170c

Include what you use in zstd_ldm_geartab

view details

Nick Terrell

commit sha 609be382acd7938b907f07162249e36702fd5b0a

Merge pull request #2719 from danlark1/danlark_iwyu Include what you use in zstd_ldm_geartab

view details

push time in 25 days