profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/eloj/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Eddy L O Jansson eloj Sweden I self-identify as a so called 'programmer'. In my spare time I enjoy the video games and stringology.

eloj/radix-sorting 13

Radix sorting from the ground up

eloj/pma 3

Memory allocation/pointer elimination experiments

eloj/crc32c 1

Example of using SSE4.2 CRC32c hardware instructions

eloj/fuzzy-search 1

Experiments in fuzzy text searching

eloj/binary-search-kdf 0

Experiment with LUT'ed binary-search

eloj/countdown 0

Countdown

eloj/go-eddy 0

Go function dumping ground

eloj/lz4 0

Extremely Fast Compression algorithm

eloj/lzw-eddy 0

A basic headerless LZW compressor & decompressor

issue commentValveSoftware/steam-for-linux

Shader Cache is Being Downloaded and Compiled After Every Single Reboot

this reads like the current intended behavior of the Steam client. These shader pre-cache updates were happening before, but with less visibility.

Is it intended that the shader downloads are not tagged as such in the downloader? You put in the effort to make it so the download could have a little info icon telling you what sort of update it is, these should clearly be marked as shader updates, but they are not.

A new (i) icon next to the game's title will reveal a tooltip displaying the types of content that is included in that update. Types consist of: Game Content, Downloadable Content, Workshop Content, and Shader Pre-caching. This icon only appears if the update is not solely game content. -- https://store.steampowered.com/oldnews/88497

gardotd426

comment created time in 2 days

push eventeloj/lz4

Claes Nästén

commit sha c756d90da96451e7124eefe5a4c7f937091c58f0

add -lrt on Solaris 10, required for nanosleep

view details

Eddy Jansson

commit sha 80e3e7034e84e7ff250d7f992ac7337551f800e8

Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE This is required to correctly size a static member to hold the hash table, whose size is derived from LZ4_MEMORY_USAGE.

view details

Eddy Jansson

commit sha 0a0922067fdb684c2c79169cc9b02118b905a6b9

Don't reuse state memory that's too small. Ensure that the memory block we're trying to reuse is large enough for the new state. Fixes #974

view details

Eddy Jansson

commit sha 6f541567836b48b0816f89269d8855e9f353c445

Remove ubuntu-16.04 as a test platform. The Ubuntu 16.04 environment is being removed by github on September 20, 2021. They will induce 'brownouts' starting from September 6 to get clients to move on. This change removes testing of GCC versions prior to 4.8, and clang versions prior to 3.9 Ref: https://github.com/actions/virtual-environments/issues/3287

view details

Eddy Jansson

commit sha 05245113c7b13abf61263e329f5974c5f7d8ded7

All compilers now support stdc90. With the removal of Ubuntu 16.04, all compilers in in the matrix supports c90, so remove this column from the matrix, and unconditionally test this.

view details

Eddy Jansson

commit sha b3fda2d21ed7eb94d02af65c51ba111aa5f9747a

All compilers now support stdc11. With the removal of Ubuntu 16.04, all compilers in in the matrix supports c11, so remove this column from the matrix, and unconditionally test this.

view details

Yann Collet

commit sha 27fd77e6cd7ea81cba18c2593f08084415cf5d06

Merge pull request #1014 from eloj/github-ubuntu-16.04-deprecation GitHub Ubuntu 16.04 deprecation

view details

Yann Collet

commit sha 163db1675c0cb5c4e35266e1c1f73e119227ad30

Merge pull request #1013 from eloj/relax-lz4-memory-usage-tunable Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE.

view details

Yann Collet

commit sha aeac6227d20f485adde5874fb37d2d3be94d8c10

Merge pull request #999 from pekdon/solaris-10 add -lrt on Solaris 10, required for nanosleep

view details

Yann Collet

commit sha d1d3266bc54a2990ea19b2f073cd4b838c0b5b01

fixed minor type, reported by @mwgamera

view details

Yann Collet

commit sha de45d6e0fa59e1c5dc21f611dd73f91ba04aaff1

minor doc clarification

view details

Nate

commit sha 0c782923359062d1c909b8cce52526ec95f28f6c

nit(programs/platform.h): replace Unicode character #1018 On line 83, U+2013 was replaced with U+002D in order to improve compatibility with MSVC 2019 and older compilers.

view details

Nate

commit sha 594ec829e372558ef5617bd508149468d9432acd

tests(unicode_lint.sh): create: lint source files in ./lib/ and ./programs/ for Unicode

view details

Nate

commit sha 7b4692596e2d2eb3542edbdf156d5e4fe68b5a4d

nit(tests/unicode_lint.sh): standardize failure format message

view details

Nate

commit sha e2561f20e6919a94d0be697db16fb6a3f4602b38

ci(.travis.yml): add unicode lint job

view details

Yann Collet

commit sha eb70459803b8ea1d879c011ed5c8f5d5c16d250b

Merge pull request #1019 from servusdei2018/patch-1 nit(programs/platform.h): replace Unicode character #1018

view details

Nate

commit sha 424e1d9622c557d7b94b55a31e9f91938b69d5e4

tests(tests/unicode_lint.sh): expand unicode lint to the ./tests/ directory

view details

Nate

commit sha 28eff0c77e32a59ce2c90511f50d8c7fe98c5bbb

ci(.github/workflows/ci.yml): run unicode lint under github actions

view details

Nate

commit sha 56ef1696fd771edaf9625f489e68cd548cf57c41

ci(.github/workflows/ci.yml): fix invalid yaml fixed syntax error

view details

Nate

commit sha 9a511781d6553752f1cc081ba009b7f330ad24fd

ci: pass script through sh

view details

push time in 4 days

push eventeloj/ansible

Eddy Jansson

commit sha cd7e6fa3165cfbb2b217296f869f9a86cc41eb50

Upgrade to golang 1.17.1 "go1.17.1 (released 2021-09-09) includes a security fix to the archive/zip package, as well as bug fixes to the compiler, linker, the go command, and to the crypto/rand, embed, go/types, html/template, and net/http packages."

view details

push time in 12 days

startedlennonhill/cookies-txt

started time in a month

push eventeloj/countdown

Eddy Jansson

commit sha 93d87ef9b367579dc0c0f5de57ed227cba1c47ef

Correct module name, imports. Update some package versions.

view details

push time in a month

push eventeloj/ansible

Eddy Jansson

commit sha b2d1241c08d66cb4b76f44dc5dd62cf746a5a88d

Add variable for go install path.

view details

push time in a month

push eventeloj/ansible

Eddy L O Jansson

commit sha 48643ea1b4f145bd4c403a6d58ca722b7026b3c5

Upgrade to golang 1.17 https://golang.org/doc/go1.17

view details

push time in a month

startedrobhagemans/hoard-of-bitfonts

started time in a month

Pull request review commentocornut/imgui

Add an example to use ImGui with SDL / SDL_Renderer

+// dear imgui: Renderer Backend for SDL_Renderer, with Platform Backend SDL ++// Implemented features:++// You can copy and use unmodified imgui_impl_* files in your project. See examples/ folder for examples of using this.+// If you are new to Dear ImGui, read documentation from the docs/ folder + read the top of imgui.cpp.+// Read online: https://github.com/ocornut/imgui/tree/master/docs+++// CHANGELOG+//  2021-16-03: Creation 

This date is very suspect.

1bsyl

comment created time in a month

PullRequestReviewEvent

startedenkimute/ganja.js

started time in a month

push eventeloj/ansible

Eddy L O Jansson

commit sha fd6815807e336846a947c1fbe350ef185645595b

Upgrade to golang 1.16.7 "go1.16.7 (released 2021-08-05) includes a security fix to the net/http/httputil package, as well as bug fixes to the compiler, the linker, the runtime, the go command, and the net/http package."

view details

push time in a month

delete branch eloj/lz4

delete branch : github-ubuntu-16.04-deprecation

delete time in 2 months

delete branch eloj/lz4

delete branch : relax-lz4-memory-usage-tunable

delete time in 2 months

PR opened lz4/lz4

GitHub Ubuntu 16.04 deprecation

Removes ubuntu-16.04 as a test platform.

The Ubuntu 16.04 environment is being removed by github on September 20, 2021. They will induce 'brownouts' starting from September 6 to get clients to move on.

This change removes testing of GCC versions prior to 4.8, and clang versions prior to 3.9

After this change all remaining compilers support both C90 and C11, so I removed those columns in separate commits, and made that test unconditional. If you'd rather keep either one or both, just say so and I'll rebase those commits.

Fixes a github workflows warning:

lz4 CI : .github#L1
The ubuntu-16.04 environment is deprecated and will be removed on September 20, 2021. Migrate to ubuntu-latest instead. For more details see https://github.com/actions/virtual-environments/issues/3287
+23 -33

0 comment

1 changed file

pr created time in 2 months

push eventeloj/lz4

Eddy Jansson

commit sha b3fda2d21ed7eb94d02af65c51ba111aa5f9747a

All compilers now support stdc11. With the removal of Ubuntu 16.04, all compilers in in the matrix supports c11, so remove this column from the matrix, and unconditionally test this.

view details

push time in 2 months

create barncheloj/lz4

branch : github-ubuntu-16.04-deprecation

created branch time in 2 months

pull request commentlz4/lz4

Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE.

So just so I understand this right, there is no reinit required if the states have the same type? i.e the else if (cctxPtr->lz4CtxState != ctxTypeID) should remain a conditional?

I can see some possible cleanups of the code, like using the now determined id (which could get a define or enum instead of further (cctxPtr->prefs.compressionLevel < LZ4HC_CLEVEL_MIN) comparisons.

eloj

comment created time in 2 months

push eventeloj/lz4

Eddy Jansson

commit sha 0a0922067fdb684c2c79169cc9b02118b905a6b9

Don't reuse state memory that's too small. Ensure that the memory block we're trying to reuse is large enough for the new state. Fixes #974

view details

push time in 2 months

pull request commentlz4/lz4

Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE.

I took your (@t-mat) changes (excluding the test-related ones) and force pushed them on this branch.

eloj

comment created time in 2 months

push eventeloj/lz4

Eddy Jansson

commit sha 80e3e7034e84e7ff250d7f992ac7337551f800e8

Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE This is required to correctly size a static member to hold the hash table, whose size is derived from LZ4_MEMORY_USAGE.

view details

Eddy Jansson

commit sha afe9d8635367f1c3d32cfe35a61bb6ff552e8257

Don't reuse state memory that's too small. Ensure that the memory block we're trying to reuse is large enough for the new state. Fixes #974

view details

push time in 2 months

pull request commentlz4/lz4

Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE.

Definitely do the better version of the PR. This coupling of lz4frame.c to lz4.c was one of the things I didn't like with this change, since it was clear the the issue was not actually with LZ4_MEMORY_USAGE in lz4.c, even if I didn't feel comfortable taking it all the way.

Feel free to close this PR at your convenience.

eloj

comment created time in 2 months

pull request commentlz4/lz4

Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE.

The assert isn't the best[0], but it's what I was comfortable with, and it should stop anyone from building the library with memory corruption by mistake.

[0] The size of the exponent isn't actually the issue; it just exposes a different issue, and you can argue its placing. There should probably be an assert inside LZ4_createStreamHC, similar to the one that exists in LZ4_createStream, to check that the static size is at least as large as required by the struct allocated there.

eloj

comment created time in 2 months

issue commentlz4/lz4

Conflict between LZ4_STREAMSIZE and LZ4_MEMORY_USAGE

#1013 is the best I can offer at the time. It does allow for some more flexibility in chosing LZ4_MEMORY_USAGE.

It's not much, I know.

mcrumpface

comment created time in 2 months

PR opened lz4/lz4

Define LZ4_STREAMSIZE in terms of LZ4_MEMORY_USAGE.

This is required to correctly size a static member to hold the hash table, whose size is derived from LZ4_MEMORY_USAGE.

At the same time, document that going beyond an exponent of 18 (for 256KiB) is not supported at this point in time.

Adds static assert to enforce this for now.

+8 -5

0 comment

3 changed files

pr created time in 2 months

create barncheloj/lz4

branch : relax-lz4-memory-usage-tunable

created branch time in 2 months

push eventeloj/lz4

Eddy Jansson

commit sha c1f514f3dbc22e56b2d6821f461aa058bd3104c3

Expand use of pkg-config variables. Change pkg-config generation such that the path variables, not their values, are used in the definitions of Libs and Cflags, and that $prefix is substituted into libdir and includedir iff they start with its value. This makes it easier to modify the already installed file if necessary.

view details

Eddy Jansson

commit sha eba110ad5b52dcd7f8cfcfddae03cc7fb9207781

Print target directories during 'make install'. This takes #975 to its logical conclusion.

view details

Yann Collet

commit sha 7be503964858691951320f13a9b5d28e450e3fad

Merge pull request #1011 from eloj/improve-pkgconfig Expand use of pkg-config variables.

view details

Yann Collet

commit sha 4de56b3da3f709e61301d65cf67e068bc650d9eb

Merge pull request #1012 from eloj/print-install-dirs Print target directories during 'make install'.

view details

push time in 2 months

delete branch eloj/lz4

delete branch : print-install-dirs

delete time in 2 months

delete branch eloj/lz4

delete branch : improve-pkgconfig

delete time in 2 months

issue commentlz4/lz4

Conflict between LZ4_STREAMSIZE and LZ4_MEMORY_USAGE

This sounded like some nice low-hanging fruit, so I took a look. Unfortunately I ran into problems I don't currently fully understand.

Making the obvious change (as suggested above) results in the following:

Setting MEMORY_USAGE in the range 14 (default) to 18 passes the tests. Setting MEMORY_USAGE to 12 hits a test for compression ratio in the tests, which makes sense. The problem shows itself when I try to go above 19.

At that point we hit memory corruption when running frame-test, specifically:

LZ4F_decompress_usingDict on multiple linked blocks : malloc(): corrupted top size or LZ4F_decompress_usingDict on multiple linked blocks : frametest: malloc.c:2379: sysmalloc: Assertion(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.`

So I did what you do in a situation like this; you run the test under valgrind:

LZ4F_compressFrame_usingCDict, multiple linked blocks : ==42084== Invalid write of size 8
==42084==    at 0x484296F: memset (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==42084==    by 0x411445: LZ4_initStream (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x402529: LZ4F_compressBegin_usingCDict (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x403392: LZ4F_compressFrame_usingCDict (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x43747C: basicTests (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x40125F: main (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==  Address 0x5599178 is 0 bytes after a block of size 262,200 alloc'd
==42084==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==42084==    by 0x432552: LZ4_createStreamHC (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x40249A: LZ4F_compressBegin_usingCDict (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x403392: LZ4F_compressFrame_usingCDict (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x4373E1: basicTests (in /home/eddy/dev/EXT/lz4/tests/frametest)
==42084==    by 0x40125F: main (in /home/eddy/dev/EXT/lz4/tests/frametest)

At this point my analysis is that we end up LZ4_initStream() which will clear the passed in memory, doing so assuming it's sizeof(LZ4_stream_t_internal) bytes, but what was actually allocated was a LZ4_streamHC_u, which is completely different size.

That we end up in LZ4_initStream using two different contexts seems weird to me, and suggests and underlying issue.

The allocation/init called seems based on the compression level, so the initial test failures in LZ4F_compressFrame_usingCDict, multiple linked blocks and LZ4F_compressFrame_usingCDict, multiple independent blocks can be force-worked around by adding code like:

+++ b/tests/frametest.c
@@ -614,6 +614,7 @@ int basicTests(U32 seed, double compressibility)
             memset(&cParams, 0, sizeof(cParams));
             cParams.frameInfo.blockMode = LZ4F_blockLinked;
             cParams.frameInfo.blockSizeID = LZ4F_max64KB;
+            cParams.compressionLevel = LZ4F_compressionLevel_max(); // XXX LZ4HC_CLEVEL_DEFAULT
             CHECK_V(cSizeContiguous,
                 LZ4F_compressFrame_usingCDict(cctx, compressedBuffer, outCapacity,
                                               CNBuffer, inSize,
@@ -650,6 +651,7 @@ int basicTests(U32 seed, double compressibility)
             memset(&cParams, 0, sizeof(cParams));
             cParams.frameInfo.blockMode = LZ4F_blockIndependent;
             cParams.frameInfo.blockSizeID = LZ4F_max64KB;
+            cParams.compressionLevel = LZ4F_compressionLevel_max(); // XXX

While this ensures we end up in LZ4_initStreamHC (which zeroes the correct size of state) instead of LZ4_initStream (which does not), this does not seem like the Right Thing, and anyhow that only takes us into fuzzerTests where we again crash due to the underlying issue.

I may continue poking at it, but in the meanwhile any direction would be appreciated.

mcrumpface

comment created time in 2 months