profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/shawnl/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

llvm/llvm-project 9668

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. Note: the repository does not accept github pull requests at this moment. Please submit your patches at http://reviews.llvm.org.

distcc/distcc 1460

distributed builds for C, C++ and Objective C

distcc/distcc.github.io 4

The distcc website

shawnl/command-not-found 3

command not found

shawnl/bedrock 1

Making mozilla.org awesome, one pebble at a time

shawnl/calagator 1

An event aggregator based in Portland, OR

shawnl/fortunes-wilde 1

Oscar Wilde Fortunes

shawnl/gitignore 1

A collection of useful .gitignore templates

mgerring/datafest-moneymap 0

Map of variables influencing money effectiveness in California

push eventdistcc/distcc

Alexey Sheplyakov

commit sha ef795d4fa8b04ee0227377a207514221f5b33451

update-distcc-symlinks: create symlinks in $prefix distccd looks for approved compilers in $prefix/lib/distcc. However update-distcc-symlinks always creates symlinks in /usr/lib. Thus if distcc is installed in a custom directory (`--prefix=/opt`, or `--prefix=$HOME`) distccd will refuse compilation jobs even after running `update-distcc-symlinks`. This patch adjusts `update-distcc-symlinks` so it creates symlinks in $prefix/lib/distcc. As a side effect it's possible to use `update-distcc-symlinks` without root permissions. Related: #431

view details

Alexey Sheplyakov

commit sha 7f5343988089d15c3cced2923a939ea2a9b0d81d

distccd: check for approved compilers in /usr/lib/distcc `dcc_check_compiler_whitelist` bails out if $prefix/lib/distcc directory does not exists. With this patch it keeps looking in /usr/lib/distcc instead. Closes: #431

view details

Shawn Landden

commit sha 1b9850947bc6ba9566a5388817d3cf432e3a827e

Merge pull request #433 from asheplyakov/fix-431 distccd: keep looking for approved compilers in /usr/lib/distcc ...

view details

push time in 4 days

PR merged distcc/distcc

distccd: keep looking for approved compilers in /usr/lib/distcc ...

... instead of bailing out if $prefix/lib/distcc directory does not exist

Closes: #431

+5 -8

0 comment

3 changed files

asheplyakov

pr closed time in 4 days

issue closeddistcc/distcc

Error: (dcc_check_compiler_whitelist) CRITICAL! no /usr/local/lib/distc

This error pops up on the server side(the one that runs distccd. I have two machines, running Ubuntu 18.04. distcc is compiled from source:

cd /tmp && wget https://github.com/distcc/distcc/releases/download/v3.4/distcc-3.4.tar.gz
tar xf ./distcc-3.4.tar.gz
cd distcc-3.4
./configure --without-libiberty
make all -j$(nproc)
make install;

In the make install step it prints the locations of the files to be installed

'make install' will install distcc as follows:
  man pages            /usr/local/share/man/man1
  documents            /usr/local/share/doc/distcc
  programs             /usr/local/bin
  sbin programs        /usr/local/sbin
  system configuration /usr/local/etc
  icon file            /usr/local/share/pixmaps
  application file     /usr/local/share/applications

Then to setup the masquade mode, I manually ran sudo update-distcc-symlinks. The daemon is started by distccd --daemon --allow 10.0.0.0/8 --log-file=/tmp/distccd.log --verbose.

On the client side, distcc was installed using the same way. When calling cmake .. I do see that distcc is being used as the C++ compiler

-- Check for working CXX compiler: /usr/lib/distcc/bin/c++
-- Check for working CXX compiler: /usr/lib/distcc/bin/c++ - works

... but the compilation failed with message:

[  3%] Building C object 3rdparty/ippiw/CMakeFiles/ippiw.dir/src/iw_image_filter_box.c.o
distcc[12865] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

And on the server side, the log file says:

distccd[3240] (dcc_r_file) received 253 bytes to file /tmp/distccd_77dbace9.ii
distccd[3240] (dcc_r_file_timed) 253 bytes received in 0.000322s, rate 767kB/s
distccd[3240] (dcc_set_input) changed input from "/tmp/opencv/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx" to "/tmp/distccd_77dbace9.ii"
distccd[3240] (dcc_set_input) command after: x86_64-linux-gnu-g++ -fPIE -o CMakeFiles/cmTC_2b53a.dir/testCXXCompiler.cxx.o -c /tmp/distccd_77dbace9.ii
distccd[3240] (dcc_set_output) changed output from "CMakeFiles/cmTC_2b53a.dir/testCXXCompiler.cxx.o" to "/tmp/distccd_76b6ace9.o"
distccd[3240] (dcc_set_output) command after: x86_64-linux-gnu-g++ -fPIE -o /tmp/distccd_76b6ace9.o -c /tmp/distccd_77dbace9.ii
distccd[3240] (dcc_check_compiler_masq) /usr/bin/x86_64-linux-gnu-g++ is a safe symlink to g++-7
distccd[3240] (dcc_check_compiler_whitelist) CRITICAL! no /usr/local/lib/distcc

I'm not sure why it looks for files under /usr/local/lib/distcc as it's not installed or symlinked by the script. Does anyone has any clue?

Thanks!

closed time in 4 days

t0ny-peng

push eventdistcc/distcc

Alexey Sheplyakov

commit sha 47a19b96561f42dc216a30fdb4b9e21b89d04278

Revert "Skip distributing LTO cc invocations" This reverts commit 8dacd28d888210753e9457eb31175d8e2a1c348e. The "LTO invocations are not worth distributing" statement is in general wrong. GCC generates the (target) machine code for every compilation unit even with `-flto` (so it makes sense to distribute these). And the whole program analysis is not guaranteed to be the bottleneck of the build. As a matter of fact distributing LTO compilations reduces the build time of LLVM/clang, GCC, and other programs (especially C++ ones). Closes: #428

view details

Shawn Landden

commit sha b94b751aef444e0e9e5224730891650d03d12a4d

Merge pull request #429 from asheplyakov/distribute-lto-again Revert "Skip distributing LTO cc invocations"

view details

push time in 9 days

PR merged distcc/distcc

Revert "Skip distributing LTO cc invocations"

This reverts commit 8dacd28d888210753e9457eb31175d8e2a1c348e. The "LTO invocations are not worth distributing" statement is in general wrong. GCC generates the (target) machine code for every compilation unit even with -flto (so it makes sense to distribute these). And the whole program analysis is not guaranteed to be the bottleneck of the build. As a matter of fact distributing LTO compilations reduces the build time of LLVM/clang, GCC, and other programs (especially C++ ones).

Closes: #428

+0 -3

0 comment

1 changed file

asheplyakov

pr closed time in 9 days

issue closeddistcc/distcc

distcc refuses to distribute LTO compilations for no good reason

"LTO cc invocations are not worth distributing"

In general this statement is not true. There's no simple way to predict if the distributed compilation is going to be faster. That depends on the code, compilation options, number of local and remote cores, etc.

Experiment

Hardware

  • local machine: 6 cores, 32 GB RAM
  • 3 remote machines, each 12 cores, 16 GB RAM

Software

  • compiler: GCC 10.3
  • software being compiled: GiNaC (C++ library)
  • compiler options: -O2 -g -pipe -flto=6 (run the whole program analysis on 6 local cores)
git clone git://ginac.de/ginac.git
git clone git://ginac.de/cln.git
cd ginac
ln -s ../cln cln
mkdir -p _build_local _build_distributed
cd _build_local
cmake -GNinja \
        -DCMAKE_BUILD_TYPE=RelWithDebInfo \
        -DCLN_USE_GMP=OFF \
        -DCMAKE_C_COMPILER=x86_64-linux-gnu-gcc \
        -DCMAKE_CXX_COMPILER=x86_64-linux-gnu-g++ \
        -DCMAKE_CXX_FLAGS="-O2 -g -pipe -flto=`nproc`" \
        -DCMAKE_C_FLAGS="-O2 -g -pipe -flto=`nproc`" \
        -DCMAKE_EXE_LINKER_FLAGS="-flto=`nproc` -O2" \
        -DCMAKE_SHARED_LINKER_FLAGS="-flto=`nproc` -O2" \
        ..
cd ../build_distributed
cmake -GNinja \
        -DCMAKE_BUILD_TYPE=RelWithDebInfo \
        -DCLN_USE_GMP=OFF \
        -DCMAKE_C_COMPILER=x86_64-linux-gnu-gcc \
        -DCMAKE_CXX_COMPILER=x86_64-linux-gnu-g++ \
        -DCMAKE_CXX_FLAGS="-O2 -g -pipe -flto=`nproc`" \
        -DCMAKE_C_FLAGS="-O2 -g -pipe -flto=`nproc`" \
        -DCMAKE_EXE_LINKER_FLAGS="-flto=`nproc` -O2" \
        -DCMAKE_SHARED_LINKER_FLAGS="-flto=`nproc` -O2" \
        -DCMAKE_CXX_COMPILER_LAUNCHER=/opt/distcc/bin/distcc \
        -DCMAKE_C_COMPILER_LAUNCHER=/opt/distcc/bin/distcc \
        ..
cd ..
time cmake --build _build_local
time cmake --build _build_distributed --parallel 36

Results

Local compilation:

328.04user 27.44system 1:06.54elapsed 534%CPU (0avgtext+0avgdata 275824maxresident)k

Distributed compilation:

134.20user 16.07system 0:34.26elapsed 438%CPU (0avgtext+0avgdata 221924maxresident)k

The distributed compilation is 2x faster. Distributed/local ratio is even higher for llvm/clang and GCC cross-toolchains.

closed time in 9 days

asheplyakov

issue commentdistcc/distcc

distcc fails to find the native compiler on rpm based distros

This is all really ugly. I know that I am the one that added this cross-compile stuff, but GCC is really at fault here. When I met Andrew Pinsky the inferiorness in cross-compile compared to clang was admitted, but no-body has come forward with a canonicalization, and done the work.

I am not sure what distcc can do, as we are kind of at the butt end of the problem.

asheplyakov

comment created time in 10 days

push eventdistcc/distcc

Alexey Sheplyakov

commit sha 879b71d6e95673e58d33f6c3c341a893ee307161

dcc_gcc_rewrite_fqn: avoid heap corruption On ALT Linux I've run into the following bug: distcc gcc -Wall -std=gnu89 -I. -O2 -o hello.o -c hello.c free(): invalid next size (fast) Aborted (core dumped) Apparently dcc_gcc_rewrite writes beyond the allocated memory: valgrind --leak-check=full -v ./distcc gcc -Wall -std=gnu89 -I. -O2 -o hello.o -c hello.c ==11382== ERROR SUMMARY: 53 errors from 5 contexts (suppressed: 0 from 0) ==11382== ==11382== 1 errors in context 1 of 5: ==11382== Invalid write of size 1 ==11382== at 0x4C349D8: strcat (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D165: dcc_gcc_rewrite_fqn (compile.c:611) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352) ==11382== Address 0x544e828 is 1 bytes after a block of size 23 alloc'd ==11382== at 0x4C31B0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D087: dcc_gcc_rewrite_fqn (compile.c:588) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352) ==11382== ==11382== ==11382== 1 errors in context 2 of 5: ==11382== Invalid write of size 1 ==11382== at 0x4C349C8: strcat (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D165: dcc_gcc_rewrite_fqn (compile.c:611) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352) ==11382== Address 0x544e827 is 0 bytes after a block of size 23 alloc'd ==11382== at 0x4C31B0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D087: dcc_gcc_rewrite_fqn (compile.c:588) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352) and ALT Linux' hardened glibc does not quite like that. Correctly compute the `newcmd_len` to avoid the problem. ALTBUG: #40425

view details

Shawn Landden

commit sha 21da64e45db5d229dbb02533274bef84702d65a1

Merge pull request #425 from asheplyakov/heapcorruption-fixup dcc_gcc_rewrite_fqn: increase buffer size by one, hopefully avoiding heap corruption

view details

push time in 10 days

PR merged distcc/distcc

dcc_gcc_rewrite_fqn: avoid heap corruption

On ALT Linux I've run into the following bug:

distcc gcc -Wall -std=gnu89 -I. -O2 -o hello.o -c hello.c free(): invalid next size (fast) Aborted (core dumped)

Apparently dcc_gcc_rewrite writes beyond the allocated memory:

valgrind --leak-check=full -v ./distcc gcc -Wall -std=gnu89 -I. -O2 -o hello.o -c hello.c

==11382== ERROR SUMMARY: 53 errors from 5 contexts (suppressed: 0 from 0) ==11382== ==11382== 1 errors in context 1 of 5: ==11382== Invalid write of size 1 ==11382== at 0x4C349D8: strcat (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D165: dcc_gcc_rewrite_fqn (compile.c:611) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352) ==11382== Address 0x544e828 is 1 bytes after a block of size 23 alloc'd ==11382== at 0x4C31B0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D087: dcc_gcc_rewrite_fqn (compile.c:588) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352) ==11382== ==11382== ==11382== 1 errors in context 2 of 5: ==11382== Invalid write of size 1 ==11382== at 0x4C349C8: strcat (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D165: dcc_gcc_rewrite_fqn (compile.c:611) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352) ==11382== Address 0x544e827 is 0 bytes after a block of size 23 alloc'd ==11382== at 0x4C31B0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==11382== by 0x10D087: dcc_gcc_rewrite_fqn (compile.c:588) ==11382== by 0x10D4B4: dcc_build_somewhere (compile.c:725) ==11382== by 0x10DC01: dcc_build_somewhere_timed (compile.c:1014) ==11382== by 0x10E380: main (distcc.c:352)

and ALT Linux' hardened glibc does not quite like that. Correctly compute the newcmd_len to avoid the problem.

ALTBUG: #40425

+1 -1

0 comment

1 changed file

asheplyakov

pr closed time in 10 days

issue commentccache/ccache

Support kyototycoon backend

El mié., 14 jul. 2021 21:58, Anders Björklund ***@***.***> escribió:

Looks like Kyoto Tycoon was abandoned again in 2017 (after being left since 2012) ? There are some minor bug fixes for Kyoto Cabinet, I guess I will add those to the others...

Since it is not available as packages in any distribution, that also makes adoption less likely.

This could become a problem for ccache, which is looking for authentication (while optional)

Kyoto Tycoon's security model is based on serving clients on a internal network as, by design, it provides no authentication mechanism.


While the storage backend "works", I'm not sure if it adds anything over Http and Redis ?

So far it seems that both the protocol (HTTP) and the database (KC) is somewhat slower. That is, for performance it is better to use Redis and RocksDB*. Otherwise, NGINX/DAV.

Yes, http and dav are great protocols, and the Kyoto libre family of software is a great backend for that. Memcached is also a great protocol. If you want security I think wireguard would be a reasonable way to get it, but that doesn't belong in ccache, but we could help the user set it up.

  • of course one could also go for (not open-source) Redis Enterprise and Redis on Flash

I think you answered your own question here.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccache/ccache/issues/435#issuecomment-880095875, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4W4UBZKLKB2QRHXYPOR3TXXF4HANCNFSM4HXTHQGA .

shawnl

comment created time in 10 days

issue commentccache/ccache

Support kyototycoon backend

All I am saying about NFS is that it is a protocol that comes out of Sun Microsystems, and it was designed by committee and sucks ass.

El lun., 12 jul. 2021 20:00, Anders Björklund ***@***.***> escribió:

The main reason for replacing memcached protocol with redis was that it allows values bigger than 1M. The client is also easier to work with, but it is expected that the overall performance is slightly lower...

It remains to be seen, having implementations available means that benchmarks can be performed ? Note that you don't have to use NFS (any network file system would do), just that it is a common choice.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccache/ccache/issues/435#issuecomment-878399365, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4W4WVSFX7ZPB4LMLCMK3TXMGTFANCNFSM4HXTHQGA .

shawnl

comment created time in 13 days

issue commentccache/ccache

Support kyototycoon backend

El lun., 12 jul. 2021 19:50, Anders Björklund ***@***.***> escribió:

Added a draft, for comparison... Not sure it is noteworthy/available enough, for inclusion ?

But it could be interesting to run some benchmarks, to see how it compares with the others.

It would be a complement to the existing backends:

  • storage::secondary::FileStorage (such as NFS)
  • storage::secondary::HttpStorage (such as NGINX)

I think this is the only needed backend. Http is a good protocol (HTTP/1.1 that is). Memcached is also a reasonable protocol, but HTTP is suitable and the others are shit. Especially NFS is a horrible protocol. Not too familiar with redis.

  • storage::secondary::RedisStorage (such as Redis)

Replacing the primary storage has not been discussed or considered, it is file-based for now.

Some older comparisons between databases are available here: http://www.lmdb.tech/bench/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccache/ccache/issues/435#issuecomment-878391408, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4W4QE2BHPL5A6CKMD4WLTXMFLHANCNFSM4HXTHQGA .

shawnl

comment created time in 13 days

issue commentccache/ccache

Support kyototycoon backend

There is a successor to Kyoto cabinet now that should be used, but I would highly recommend that over redis. These databases are extremely solid, so much so that Kyoto cabinet only had to release once after a few years because of a new version of GCC giving a warning (because C++ is a meh language), but otherwise it was just so stable compared to all those VC fast fast fast people.

El dom., 11 jul. 2021 20:44, Anders Björklund ***@***.***> escribió:

The new API is completed, so there is nothing stopping doing an implementation also for Kyoto Tycoon/Cabinet...

Seems that it now lives at https://dbmx.net/kyototycoon/ and https://dbmx.net/kyotocabinet/ (decendent of FAL Labs)

However, it seems that when using the Redis backend the same features can be achieved - but more supported ?

Using "Ardb https://github.com/yinqiwen/ardb" provides a similar mapping as KT, using redis (instead of memcached) and RocksDB (instead of KC).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccache/ccache/issues/435#issuecomment-877830382, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4W4Q2QQXZ7ATPWCAOCXLTXHC67ANCNFSM4HXTHQGA .

shawnl

comment created time in 14 days

issue commentziglang/zig

stage1 C ABI compatibility

It has been mentioned on the LLVM mailing list, but LLVM is the place where the C ABI needs to be implemented, instead of the hacked together interaction between clang and LLVM.

El jue., 8 jul. 2021 15:37, Exonorid ***@***.***> escribió:

Vulkan's debug utilities use a callback of the type fn (VkDebugUtilsMessageSeverityFlagBitsEXT, VkDebugUtilsMessageTypeFlagsEXT, *const VkDebugUtilsMessengerCallbackDataEXT, ?*c_void) callconv(.C) u32 which doesn't compile

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ziglang/zig/issues/1481#issuecomment-876366047, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4W4U5MDWYJ2T45UOF7ULTWWEVZANCNFSM4FTUZZGA .

andrewrk

comment created time in 17 days

startedvnmakarov/mir

started time in a month

pull request commentdistcc/distcc

Skip distributing LTO cc invocations

@asheplyakov

I didn't put much thought into this before merging it. There are also things like ThinLTO, and I am not really up on everything.

What behavior do you think distcc should have? What about throwing an error when called with -flto?

RX14

comment created time in a month

push eventdistcc/distcc

hudeng

commit sha b35c8914d9b1e048b8a13e7cdeefe8bc44b4d435

fix: make uninstall failed

view details

Shawn Landden

commit sha 68a9d020737449a6bc6cb09cf7161caa47dc498d

Merge pull request #424 from hudeng-go/master fix: make uninstall failed Inserts missing semicolon

view details

push time in a month

PR merged distcc/distcc

fix: make uninstall failed

Solve the problem of failure to execute "make uninstall"

+1 -1

0 comment

1 changed file

hudeng-go

pr closed time in a month

issue commenttermux/termux-packages

Package request: Avahi

El lun., 7 jun. 2021 12:17, Abolish ICE ***@***.***> escribió:

If you are using the Android platform you should probably use their implementation, otherwise you should use some other platform, and not something as hacky as Termux.

Absolutely! So: how? The "implementation" appears to be nothing but raw API functions, without any apps to drive them.

You just stated the Free Software manifesto.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/termux/termux-packages/issues/65#issuecomment-855710673, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4W4SQWMEH2FRN45FK4TLTRR6CBANCNFSM4BVITXHQ .

regagain

comment created time in 2 months

issue commentdistcc/distcc

make deb fails

the make deb / make rpm stuff is not supported and I should remove it when i get the chance

Ntemis

comment created time in 2 months

issue commentdistcc/distcc

make check failed about distcc@3.3.3

You should never compile packages as root, this is not supported.

Tom-python0121

comment created time in 2 months

issue closeddistcc/distcc

make check failed about distcc@3.3.3

I meet a problem:make check failed about distcc@3.3.3

see:

[root@centos8 distcc]# make check
……
3.6.8 (default, Aug 24 2020, 18:22:40)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

CompileHello_Case              FAIL
-----------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/spack-stage/spack-stage-distcc-3.3.3-6xazc2juijrqqxj5vdblogdwlxq4kfoa/spack-src/test/comfychair.py", line 348, in runtest
    obj.setup()
  File "./test/testdistcc.py", line 881, in setup
  File "./test/testdistcc.py", line 312, in setup
  File "./test/testdistcc.py", line 383, in startDaemon
  File "/tmp/root/spack-stage/spack-stage-distcc-3.3.3-6xazc2juijrqqxj5vdblogdwlxq4kfoa/spack-src/test/comfychair.py", line 122, in fail
    raise AssertionError(reason)
AssertionError: failed to start daemon: 107
test_log:
Run command: /bin/cc --version
Wait status: 0x0 (exit code 0, signal 0)
stdout:
cc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


stderr:
Run command: /bin/cc --version
Wait status: 0x0 (exit code 0, signal 0)
stdout:
cc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


stderr:
Run command: distccd --verbose --lifetime=60 --daemon --log-file '/tmp/root/spack-stage/spack-stage-distcc-3.3.3-6xazc2juijrqqxj5vdblogdwlxq4kfoa/spack-src/_testtmp/CompileHello_Case/distccd.log' --pid-file '/tmp/root/spack-stage/spack-stage-distcc-3.3.3-6xazc2juijrqqxj5vdblogdwlxq4kfoa/spack-src/_testtmp/CompileHello_Case/daemonpid.tmp' --port 42000 --allow 127.0.0.1 --enable-tcp-insecure
Wait status: 0x6b00 (exit code 107, signal 0)
stdout:

stderr:
distccd[70683] (dcc_preferred_user) Warning: no such user as "distcc"
distccd[70683] (dcc_discard_root) successfully set no_new_privs
distccd[70683] (dcc_discard_root) discarded root privileges, changed to uid=65534 gid=65534
distccd[70683] (dcc_setup_real_log) ERROR: failed to open /tmp/root/spack-stage/spack-stage-distcc-3.3.3-6xazc2juijrqqxj5vdblogdwlxq4kfoa/spack-src/_testtmp/CompileHello_Case/distccd.log: Permission denied

-----------------------------------------------------------------
CommaInFilename_Case           FAIL
-----------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/spack-stage/spack-stage-distcc-3.3.3-6xazc2juijrqqxj5vdblogdwlxq4kfoa/spack-src/test/comfychair.py", line 348, in runtest
    obj.setup()
  File "./test/testdistcc.py", line 881, in setup
  File "./test/testdistcc.py", line 312, in setup
  File "./test/testdistcc.py", line 383, in startDaemon
  File "/tmp/root/spack-stage/spack-stage-distcc-3.3.3-6xazc2juijrqqxj5vdblogdwlxq4kfoa/spack-src/test/comfychair.py", line 122, in fail
    raise AssertionError(reason)
AssertionError: failed to start daemon: 107
test_log:
Run command: /bin/cc --version
Wait status: 0x0 (exit code 0, signal 0)
stdout:
cc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


stderr:
Run command: /bin/cc --version
Wait status: 0x0 (exit code 0, signal 0)
stdout:
cc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I want to know what I can do to avoid a make check failure.

Information on your system

[root@centos8 spack-src]# spack debug report

  • Spack: 0.16.1-1624-a0b5dcca3c
  • Python: 3.6.8
  • Platform: linux-centos8-aarch64
  • Concretizer: original

closed time in 2 months

Tom-python0121

release distcc/distcc

v3.4

released time in 2 months

created tagdistcc/distcc

tagv3.4

distributed builds for C, C++ and Objective C

created time in 2 months

push eventdistcc/distcc

Shawn Landden

commit sha 50d821efe99cae82c05be0a4ab3b4035ef0d3883

3.4

view details

push time in 2 months

PR merged distcc/distcc

parse_command: handle Xclang correctly
+1 -0

0 comment

1 changed file

ddcc

pr closed time in 2 months

push eventdistcc/distcc

Dominic Chen

commit sha 750dd02b9af9b43dd372d63841ff7552f1194507

parse_command: handle Xclang correctly

view details

Shawn Landden

commit sha eadc3800906c9487ca1a1461a511ac372826e06b

Merge pull request #419 from ddcc/master parse_command: pass through Xclang argument

view details

push time in 2 months

push eventdistcc/distcc

Alexey Sheplyakov

commit sha ac121434aae7d16053d005c508073bf1a8108095

prefork: use available cores more efficiently Observed behavior ----------------- When compiling a big C project (Linux kernel) with distcc compilation nodes never use 100% of CPU time, instead a system is 25 -- 40% idle (despite --jobs is set to 3x the number of cores). Analysis -------- Linux kernel consists of many relatively small C sources and header files. Compiling a single source file typically takes a fraction of a second (although there are quite a number of "heavier" sources). `dcc_preforked_child` exits after 50 requests. Thus a typical lifetime of a worker process is just a few seconds. However `dcc_create_kids` sleeps for a second after every fork. Also `dcc_preforked_child` sleeps for yet another second. Which is >= 25% of a typical child lifetime. As a result distccd is unable to use available CPU time (system load is always <= 75% during the compilation). Fix --- Removed `sleep` from `dcc_preforking_parent` and `dcc_create_kids`. Also made `dcc_preforked_child` process compilation jobs for at least 60 seconds. Result ------ Kernel compilation time (two 6-core nodes) reduced almost 2x: before: 20 minutes, after: 12 minutes Closes: #420

view details

Shawn Landden

commit sha c7337a78a248b557463f0fe281659efdfe661134

Merge pull request #421 from asheplyakov/distccd-dontsleep prefork: use available cores more efficiently

view details

push time in 2 months

PR merged distcc/distcc

prefork: use available cores more efficiently

Observed behavior

When compiling a big C project (Linux kernel) with distcc compilation nodes never use 100% of CPU time, instead a system is 25 -- 40% idle (despite --jobs is set to 3x the number of cores).

Analysis

Linux kernel consists of many relatively small C sources and header files. Compiling a single source file typically takes a fraction of a second (although there are quite a number of "heavier" sources). dcc_preforked_child exits after 50 requests. Thus a typical lifetime of a worker process is just a few seconds. However dcc_create_kids sleeps for a second after every fork. Also dcc_preforked_child sleeps for yet another second. Which is >= 25% of a typical child lifetime. As a result distccd is unable to use available CPU time (system load is always <= 75% during the compilation).

Fix

Removed sleep from dcc_preforking_parent and dcc_create_kids. Also made dcc_preforked_child process compilation jobs for at least 60 seconds.

Result

Kernel compilation time (two 6-core nodes) reduced almost 2x: before: 20 minutes, after: 12 minutes

Closes: #420

+7 -10

1 comment

1 changed file

asheplyakov

pr closed time in 2 months