profile
viewpoint

weisslj/cpp-argparse 73

NOT YET: Python's excellent ArgumentParser in C++

weisslj/cpp-optparse 70

Python's excellent OptionParser in C++

weisslj/cmus_lastfm_autoadd 15

Automatically fill cmus playlist with new similar tracks

weisslj/clipperz-communityEdition 6

Clipperz Community Edition

weisslj/dir-edit 6

Rename or remove files in a directory using an editor

weisslj/fingerprint-store 6

List, remove or store fingerprinting track ids in your music files

weisslj/choose-next 5

Chooses a file from a directory. Very handy to re-watch tv series!

weisslj/cmuscrobbler 1

Scrobble your Songs that you listened to in Cmus

weisslj/gen-album-views 1

Generate different views of your music collection via hard links

pull request commentelixir-lang/elixir

Make ExUnit server timeout configurable

Oh, those are all in Elixir's test suite itself. I see. Those tests should definitely not take more than 60 seconds, so I would say there is something wrong going on. But yeah, you would have to change the test_helper.exs by hand... we can provide an environemnt variable, but I am afraid it may not solve the issue.

JGulbronson

comment created time in 6 minutes

pull request commentelixir-lang/elixir

Make ExUnit server timeout configurable

None of those options seems to apply directly to "make test". Looking at Makefile I don't see a way to pass global timeout value from environment or something. I am trying to avoid modifying the code directly to increase the timeouts.

JGulbronson

comment created time in 23 minutes

pull request commentelixir-lang/elixir

Make ExUnit server timeout configurable

Yes, if you look at the error message printed on each test it should tell you exactly how to configure it. :)

JGulbronson

comment created time in 34 minutes

pull request commentelixir-lang/elixir

Make ExUnit server timeout configurable

Is there a way to set global timeout before I call "make test"? I am hitting "** (ExUnit.TimeoutError) test timed out after 60000ms." with 1.11.2.

  1) test in/2 too large list in guards (KernelTest)
     test/elixir/kernel_test.exs:429
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

  2) test properly formats errors (Kernel.CLITest)
     test/elixir/kernel/cli_test.exs:46
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

  3) test properly formats errors (Kernel.CLI.RPCTest)
     test/elixir/kernel/cli_test.exs:85
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

  4) test encode then decode is identity (BaseTest)
     test/elixir/base_test.exs:783
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

  5) test defguard(p) compilation fails on expressions not allowed in guards (Kernel.GuardTest)
     test/elixir/kernel/guard_test.exs:272
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:
JGulbronson

comment created time in an hour

pull request commenterlang/otp

Add data_only option to binary_to_term

Second, I disagree with placing the filtering here; the bug if any lies in Elixir or application code blindly executing stuff received from the outside world, preventing transport of such stuff is just a workaround for that bug.

Although we are working on fixing the root cause, in practice you want to add as many safety nets as you can. If you are certain you should only receive certain terms, then being able to forbidden certain entries, as proposed here, is very valuable.

For example, in the case linked above, I would say they should also have signed the payload. If they signed it and listed only the terms they expected, they would not have been hit by the Elixir bug. If there is ever a bug in binary_to_term, the other two protections would have helped them. If they sign/encrypt it and their key leaks, then they have other two layers, and so on.

The filtering can also be done after calling binary_to_term, but I think having a higher-level API that makes it easy can be a net positive. :) I agree with the comments about data_only not being ideal. There is also an option for explicitly listing the terms you expect (instead of forbidding the ones you don't want).

michalmuskala

comment created time in an hour

pull request commenterlang/otp

Add data_only option to binary_to_term

I agree with @okeuday 's comment above. First, data_only as a name doesn't match what it does (all the things it rejects are data). Second, I disagree with placing the filtering here; the bug if any lies in Elixir or application code blindly executing stuff received from the outside world, preventing transport of such stuff is just a workaround for that bug. Third, the correct way of preventing receipt of such stuff is to add negative filters to binary_to_term/2, for instance I can see the value in preventing creation of new atoms. As for the proposed rejection of references, I honestly don't see the point of that (a reference doesn't do anything, and is isomorphic to a unique integer).

michalmuskala

comment created time in 2 hours

pull request commenterlang/otp

Support new instructions in HiPE

Given that HiPE has been removed from master, I think this should be declined.

michael-swan

comment created time in 3 hours

PR closed erlang/otp

Reviewers
Do not centralize type/cname lookups through inet_db team:PS waiting

Before this patch, inet_res:getbyname by name lookups were centralized through the inet_db process, which made it a bottleneck on concurrent machines that need to establish many connections to several hosts quickly.

Given these lookups are concurrent, the update of resolve.conf has been changed to use atomic operations that replace the list of servers and list of names.

The inet_db API has also been cleaned up to no longer expose operations that are not used anywhere within OTP, especially to avoid del+ins/add operations which wouldn't be safe concurrently.

+28 -70

12 comments

1 changed file

josevalim

pr closed time in 6 hours

pull request commenterlang/otp

Do not centralize type/cname lookups through inet_db

@RaimoNiskanen I have sent PR #2891 that moves it to a set. I will close this one for now and resubmit it once #2891 is merged.

josevalim

comment created time in 6 hours

PR opened erlang/otp

Convert inet_cache to a set

This patch is just a refactoring but it should provide faster operations at the cost of slightly higher storage (which is acceptable since the table is relatively small). This will also make it possible to have decentralized lookups in the future.

This is prep-work for the PR in #2818.

+47 -55

0 comment

1 changed file

pr created time in 6 hours

pull request commenterlang/otp

Initial implementation of EEP 50

@jhogberg I have updated the PR to support a version flag. Docs have been updated too.

I have also benchmarked maps:intersect/2 and it is slower then the current cerl_sets implementation. You can see the benchmark here and the benchmark results here. For this reason, I decided to keep the cerl_sets implementation for intersection for now. If you want to give it a try yourself, you can clone the repo, run mix deps.get and then mix run bench/intersection.exs.

josevalim

comment created time in 8 hours

PR opened erlang/otp

Support IEEE 754-2008 16-bits floats in bitstrings

This format is often used with computer graphics and AI, also in cases where data is large enough that the decreased bandwidth and storage compensates the loss of precision.

Recent compilers and platforms provide support for _Float16. Recent CPUs provide specific instruction sets for them too. In some, _Float16 is storage only, which is enough for our use cases. If _Float16 is not available, functions for conversion from float to half-floats are used, ported from https://github.com/Maratyszcza/FP16.

I am sending this pull request to collect feedback. I will work on a complete test suite if there is interest in moving this forward.

+283 -13

0 comment

4 changed files

pr created time in 9 hours

pull request commenterlang/otp

Make the Erlang installation directory relocatable on Unix-like systems

Hello Kjell,

I have already merged the first commit in this pull-request to the master branch. Can you remove the first commit and rebase the second commit on master?

Here is a second revision with the (already merged) first commit removed and the second one rebased on master.

We use lower_case_with_underscores for variable names in most shell scripts. Can you change the variable names so they use this naming convention?

I've switched to the lower_case_with_underscores convention for script variable names.

I think it would be appropriate if you add a comment with a reference to the dx code.

I've also added a comment making a reference to the dx script as suggested.

It would be great if we could add an automated test for the symlink traversal code. That way we could see if something breaks in any of the machines in Erlang/OTP's testing infrastructure.

I'm not familiar at all with the test suite, I'll need much more time to investigate it and see how a test could fit into erlexec_SUITE.erl.

A testcase could check that one can execute different types of symlinks (one or more jumps, relative paths and absolute paths) that points to the erl script.

Please let me know if you need help with anything related to testing.

Do you have a suggestion about how to create the symlinks on the fly as part of one or more tests, especially the ones with an absolute path? Could you share a small sample code maybe?

Thanks, Jérôme

JeromeDeBretagne

comment created time in 18 hours

startedhrissan/schwidko

started time in 18 hours

PR opened erlang/otp

Remove potential lag from mnesia_log:log/slog functions

This PR addresses the very rare but unfortunately existing problem with occassional latencies (up to 250ms) in mnesia operations described in more detail here: https://bugs.erlang.org/browse/ERL-1423

+4 -1

0 comment

1 changed file

pr created time in 19 hours

issue commentelixir-lang/elixir

Support Erlang/OTP 23 new features

@lukaszsamson not really because it would load from applications you don't depend on. It could be good for discovery but it can end-up bringing all of OTP, which perhaps is a bit too much.

josevalim

comment created time in a day

issue commentelixir-lang/elixir

Support Erlang/OTP 23 new features

Wouldn't :code.all_available/0 make sense in iex autocomplete? https://github.com/elixir-lang/elixir/blob/660ac71d0eccbe9c7424e708e36493d6cecadc6b/lib/iex/lib/iex/autocomplete.ex#L429

josevalim

comment created time in a day

pull request commenterlang/otp

Make ct a behaviour for test suites

@paulo-ferraz-oliveira Thanks for taking over. I've had trouble keeping up with things.

richcarl

comment created time in a day

pull request commenterlang/otp

Make the Erlang installation directory relocatable on Unix-like systems

It would be great if we could add an automated test for the symlink traversal code. That way we could see if something breaks in any of the machines in the Erlang/OTP's testing infrastructure.

A good place to add such a test case would be in this test suite:

$ERL_TOP/erts/test/erlexec_SUITE.erl

You can check if the test is running on a UNIX system similarly to how it is done on line 287 in erlexec_SUITE.erl.

A testcase could check that one can execute different types of symlinks (one or more jumps, relative paths and absolute paths) that points to the erl script.

This document describes how to run test cases:

https://github.com/erlang/otp/blob/master/HOWTO/TESTING.md

Please let me know if you need help with anything related to testing.

JeromeDeBretagne

comment created time in a day

pull request commenterlang/otp

fix crypto hmac performance loss

There are two sources to the speed-up. One is a fix of a lock problem, the other is to use HMAC low-level functions including a fix of that code with a 2X speedup. The flag only affects whether to use low-level or not. --disable-evp-hmac enables the low-level functions, and it is always automatically added for OpenSSL versions < 1.0.2. If, and how much the lock fix improves depends on the OpenSSL version, the processor architecture, number of cores and mostly by the load. The reason to keep the EVP variant is that EVP makes it possible to use HW-acceleration. With the low-level functions that is not possible. Today is only version 1.1.1 maintained with security fixes by OpenSSL, and 3.0 is on its way out. From 3.0 the low-level routines are deprecated so if we removed the EVP now we would have been forced to add them again in a couple of years.

mikpe

comment created time in a day

Pull request review commenterlang/otp

Make the Erlang installation directory relocatable on Unix-like systems

 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.-# +# # %CopyrightEnd% #-ROOTDIR="%FINAL_ROOTDIR%"+if [ -z "$ROOTDIR" ]+then+    ROOTDIR="%FINAL_ROOTDIR%"+fi++# If ROOTDIR is not set or if it doesn't point to an existing directory, try+# to find the path of the script from within itself to set ROOTDIR dynamically+if [ -z "$ROOTDIR" -o ! -d "$ROOTDIR" ]+then+    prog="$0"+    # Follow all symlinks, whether relative or absolute, leading to the script+    while [ -h "${prog}" ]; do

Great suggestion, I'll do this in my next revision

JeromeDeBretagne

comment created time in a day

Pull request review commenterlang/otp

Make the Erlang installation directory relocatable on Unix-like systems

 # if [ -z "$ROOTDIR" ] then-   ROOTDIR=%FINAL_ROOTDIR%+    ROOTDIR=%FINAL_ROOTDIR% fi  if [ -z "$RELDIR" ] then-   RELDIR=$ROOTDIR/releases+    RELDIR=$ROOTDIR/releases+fi++# If ROOTDIR is not set or if it doesn't point to an existing directory, try+# to find the path of the script from within itself to set ROOTDIR dynamically+if [ -z "$ROOTDIR" -o ! -d "$ROOTDIR" ]+then+    prog="$0"+    # Follow all symlinks, whether relative or absolute, leading to the script+    while [ -h "${prog}" ]; do+        newProg=`ls -ld "${prog}"`+        newProg=`expr "${newProg}" : ".* -> \(.*\)$"`+        if expr "x${newProg}" : 'x/' >/dev/null; then

Hi Kjell,

Indeed, this checks if newProg starts with a leading / character via a regular expression match:

$ expr "x/test" : 'x/' > /dev/null
$ echo $?
0

$ expr "xtest" : 'x/' > /dev/null
$ echo $?
1

=> the expression returns 0 if there is a match, 1 otherwise.

P.S. The returned output is the number of characters that matched, but this is not relevent hence the >/dev/null part

$ expr "x/test" : 'x/'
2

In the above case, it shows that the first 2 characters of "x/test" have matched with 'x/'

JeromeDeBretagne

comment created time in a day

PR opened erlang/otp

Add support of new cipher suite for tls1.2

Change-Id: Id512c6b6434e7fbcd6f20ffe7819e6a03c6eaf93

For ERL-1318

+144 -6

0 comment

5 changed files

pr created time in a day

pull request commenterlang/otp

fix crypto hmac performance loss

Are all changes only relevant when EVP HMAC is disabled? For example isn't the 2X slowdown relevant for EVP HMAC?

mikpe

comment created time in a day

push eventerlang/otp

Mikael Pettersson

commit sha 7e2383ad3546505632c047ddff04c5bcbb9b6df9

make_driver_tab: also strip .gprof marker from static archive Otherwise the build fails due to invalid identifiers.

view details

Mikael Pettersson

commit sha 9af0fa871420bbc4656f7f6b27e3c9dbb21dbfef

crypto: enable gprof build

view details

Hans Nilsson

commit sha b32ad88bd40ecdfd7262c75b78ab124ca34ac193

Merge pull request #2876 from mikpe/crypto-static-and-gprof Allow building VM with static and gprof-enabled crypto NIF OTP-17029

view details

Hans Nilsson

commit sha 1bb9e8d09470eda6162d02a2b340f620fd496cfe

Merge branch 'maint' * maint: crypto: enable gprof build make_driver_tab: also strip .gprof marker from static archive

view details

push time in a day

pull request commenterlang/otp

Allow building VM with static and gprof-enabled crypto NIF

Thanks for the PR!

mikpe

comment created time in a day

push eventerlang/otp

Mikael Pettersson

commit sha 7e2383ad3546505632c047ddff04c5bcbb9b6df9

make_driver_tab: also strip .gprof marker from static archive Otherwise the build fails due to invalid identifiers.

view details

Mikael Pettersson

commit sha 9af0fa871420bbc4656f7f6b27e3c9dbb21dbfef

crypto: enable gprof build

view details

Hans Nilsson

commit sha b32ad88bd40ecdfd7262c75b78ab124ca34ac193

Merge pull request #2876 from mikpe/crypto-static-and-gprof Allow building VM with static and gprof-enabled crypto NIF OTP-17029

view details

push time in a day

PR merged erlang/otp

Allow building VM with static and gprof-enabled crypto NIF fix team:PS testing

While investigating the crypto hmac performance loss in ERL-1400 I needed to build with crypto and openssl gprof-enabled and statically linked into the VM. That didn't work due to two separate build issues, fixed by this series.

+8 -1

1 comment

2 changed files

mikpe

pr closed time in a day

PR closed erlang/otp

fix crypto hmac performance loss team:PS

Fixes the crypto hmac performance loss reported in ERL-1400.

When upgrading from OTP-21 to OTP-22/23, the crypto hmac calls change from using openssl's low-level HMAC() interface to its higher-level EVP_PKEY interface. With openssl-1.0.x (as found in some enterprise environments), this causes each hmac call to require several reference count changes in openssl as pkey objects are accessed and released. openssl-1.0.x allows the client application to register an add_lock callback to atomically add to an integer in memory and return its new value. OTP does not register that callback, which causes openssl-1.0.x to fall back to taking and releasing a global pkey mutex in each reference count change. That mutex quickly causes exponential slowdown even with moderate concurrency. The solution is to implement and register the callback. With openssl-1.1.x this is not an issue since it implements this synchronization primitive itself.

With the above fix the exponential slowdown was cured, but we still saw a 2X slowdown compared to OTP-21. Profiling with gprof revealed that the new API has a lot of internal overheads from dynamic object allocation and deletion. A simpler interface not using EVP_PKEY is still present in crypto as hmac_low_level(), so we switched to that.

Even when using hmac_low_level() we saw a 2X slowdown compared to OTP-21, with both openssl-1.0.x and openssl-1.1.x. A comparison between the hmac_nif() in OTP-21 and hmac_low_level() in OTP-22/23 revealed the reason for that: the newer code calls HMAC() twice where the older code only called it once, which is enough since the size of the result is bounded by a small constant. Reinstating that logic in hmac_low_level() eliminated the final 2X slowdown.

+35 -27

4 comments

6 changed files

mikpe

pr closed time in a day

pull request commenterlang/otp

fix crypto hmac performance loss

Merged with an --disable-evp-hmac config option. EVP HMAC is always disabled for OpenSSL < 1.0.2 Thanks for the PR!

mikpe

comment created time in a day

more