Description of the problem / feature request:

I'm running some test of TensorFlow using bazel but on our multi-core POWER9 system it fails with e.g.

ERROR: /dev/shm/s3248973-EasyBuild/TensorFlow/2.4.0/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-r2.4/tensorflow/core/platform/BUILD:1142:11: failed (Exit 1): generate-xml.sh failed: error executing command

I.e. there is no good error message, it simply failed to execute that script which comes from the Bazel installation. I verified that the executed command (bazel -s) runs correctly and the script hence also exists

I even modified that script in the Bazel sources to print something at the start but that doesn't show up. So it seems that script is not (yet?) created when Bazel tries to execute it. I hence expect a race condition or something but am unable to verify this.

Any hints, ideas, ...?

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Sorry, only thing I have is the command I use to test TF:

bazel --output_base=/dev/shm/s3248973-EasyBuild/TensorFlow/2.4.0/fosscuda-2019b-Python-3.7.4/tmptspeEg-bazel-tf/output_base --install_base=/dev/shm/s3248973-EasyBuild/TensorFlow/2.4.0/fosscuda-2019b-Python-3.7.4/tmptspeEg-bazel-tf/output_base/inst_base --output_user_root=/dev/shm/s3248973-EasyBuild/TensorFlow/2.4.0/fosscuda-2019b-Python-3.7.4/tmptspeEg-bazel-tf/output_user_root --host_jvm_args=-Xms512m --host_jvm_args=-Xmx4096m test --compilation_mode=opt --config=opt --subcommands --verbose_failures --config=noaws --jobs=64 --copt="-fPIC"  --distinct_host_configuration=false --test_output=errors --local_test_jobs=1 --build_tests_only --test_tag_filters='-no_gpu,-no_oss,-oss_serial,-benchmark-test,-no_oss_py37,-v1only' --build_tag_filters='-no_gpu,-no_oss,-oss_serial,-benchmark-test,-no_oss_py37,-v1only'  -- //tensorflow/core/... //tensorflow/cc/... //tensorflow/c/... -//tensorflow/core:example_java_proto -//tensorflow/core/example:example_protos_closure

What operating system are you running Bazel on?

RHEL 7.6

What's the output of bazel info release?

release 3.4.1- (@non-git)

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

EXTRA_BAZEL_ARGS="--jobs=176 --host_javabase=@local_jdk//:jdk" ./compile.sh

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

ERROR: /dev/shm/s3248973-EasyBuild/TensorFlow/2.4.0/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-r2.4/tensorflow/core/platform/BUILD:1142:11:  failed (Exit 1): generate-xml.sh failed: error executing command 
  (cd /dev/shm/s3248973-EasyBuild/TensorFlow/2.4.0/fosscuda-2019b-Python-3.7.4/tmptspeEg-bazel-tf/output_base/execroot/org_tensorflow && \
  exec env - \
    PATH=/usr/bin:/bin \
> Replace these lines with your answer.
>
> If the files are large, upload as attachment or provide link.

    TEST_BINARY=tensorflow/core/platform/platform_strings_test \
    TEST_NAME=//tensorflow/core/platform:platform_strings_test \
    TEST_SHARD_INDEX=0 \
    TEST_TOTAL_SHARDS=0 \
  external/bazel_tools/tools/test/generate-xml.sh bazel-out/ppc-opt/testlogs/tensorflow/core/platform/platform_strings_test/test.log bazel-out/ppc-opt/testlogs/tensorflow/core/platform/platform_strings_test/test.xml 0 0)
Execution platform: @local_execution_config_platform//:platform

It's probably not too challenging to set up a bazel-bench pipeline to benchmark the java_tools commits. We can talk more next week if you're interested.

we might want to consider automatic releasing or at least benchmarking the commits that go into java_tools... i almost have a script for that by now

aha, the first commit brought back the System.gc that @larsrc-google recently removed and the second commit removed it again

Raw data:

RESULTS:
Bazel commit: /tmp/benchmark-4-febf4438cd_HEAD_1, Project commit: c85ecbed46d4703c7d6bf6b52dab98972dbe9ea1, Project source: None
  metric          mean                median          stddev      pval   
    wall:  163.424s             163.606s               0.718s            
     cpu:   54.550s              54.520s               1.046s            
  system:   16.302s              16.280s               0.094s            
  memory:   92.400s              92.000s               0.490s            

Bazel commit: /tmp/benchmark-4-febf4438cd_HEAD, Project commit: c85ecbed46d4703c7d6bf6b52dab98972dbe9ea1, Project source: None
  metric          mean                median          stddev      pval   
    wall:  163.633s ( +0.13%)   162.889s ( -0.44%)     1.184s    0.12698 
     cpu:   53.980s ( -1.04%)    53.710s ( -1.49%)     1.565s    0.12698 
  system:   16.528s ( +1.39%)    16.290s ( +0.06%)     0.549s    0.12698 
  memory:   92.800s ( +0.43%)    93.000s ( +1.09%)     0.400s    0.12698 
no regression

RESULTS:
Bazel commit: /tmp/benchmark-5-5c8e584e88_HEAD_1, Project commit: c85ecbed46d4703c7d6bf6b52dab98972dbe9ea1, Project source: None
  metric          mean                median          stddev      pval   
    wall:  162.196s             161.956s               1.264s            
     cpu:   53.400s              53.060s               0.952s            
  system:   16.470s              16.430s               0.294s            
  memory:   92.800s              93.000s               0.400s            

Bazel commit: /tmp/benchmark-5-5c8e584e88_HEAD, Project commit: c85ecbed46d4703c7d6bf6b52dab98972dbe9ea1, Project source: None
  metric          mean                median          stddev      pval   
    wall:  226.739s (+39.79%)   224.517s (+38.63%)     7.650s    0.99206 
     cpu:   53.824s ( +0.79%)    54.330s ( +2.39%)     1.454s    0.12698 
  system:   16.730s ( +1.58%)    16.730s ( +1.83%)     0.140s    0.64286 
  memory:   92.200s ( -0.65%)    92.000s ( -1.08%)     0.748s    0.12698 
anything else I can help you with?

RESULTS:
Bazel commit: /tmp/benchmark-8-a4251eab69_HEAD_1, Project commit: c85ecbed46d4703c7d6bf6b52dab98972dbe9ea1, Project source: https://github.com/bazelbuild/bazel.git
  metric          mean                median          stddev      pval   
    wall:  222.846s             221.188s              5.016s             
     cpu:   53.466s              54.300s              1.425s             
  system:   16.564s              16.500s              0.177s             
  memory:   92.200MB             92.000MB             0.400MB            

Bazel commit: /tmp/benchmark-8-a4251eab69_HEAD, Project commit: c85ecbed46d4703c7d6bf6b52dab98972dbe9ea1, Project source: https://github.com/bazelbuild/bazel.git
  metric          mean                median          stddev      pval   
    wall:  164.262s  (-26.29%)  162.785s  (-26.40%)   3.078s     0.99206 
     cpu:   56.008s  ( +4.75%)   55.070s  ( +1.42%)   2.373s     0.64286 
  system:   16.352s  ( -1.28%)   16.160s  ( -2.06%)   0.411s     0.64286 
  memory:   92.800MB ( +0.65%)   93.000MB ( +1.09%)   0.400MB    0.64286

Description of the problem / feature request:

Our bazel-bench pipeline runs nightly benchmarking of a sample of the bazel commits throughout the day, by using the bazel binaries built at those commits to build the Bazel project itself (bazel build //src:bazel). We've identified the following:

1. 5c92b11dc5e1decf1b2a30b72d9bdaaf6e3e4380 regressed wall time by ~48%.

Bazel commit: 5c92b11dc5e1decf1b2a30b72d9bdaaf6e3e4380, Project commit: c282526c071389cd6f88cb77565283b257316267, Project source: https://github.com/bazelbuild/bazel.git
  metric          mean                median          stddev      pval   
    wall:  116.968s ( -0.93%)   117.305s ( -0.48%)     2.287s    0.00000 
     cpu:   79.557s ( +0.29%)    80.480s ( +0.51%)     1.464s    0.40000 
  system:   19.167s ( -1.83%)    19.240s ( -2.58%)     0.148s    0.40000 
  memory:   94.667s ( +0.00%)    95.000s ( +0.00%)     0.471s    0.00000 

Bazel commit: c282526c071389cd6f88cb77565283b257316267, Project commit: c282526c071389cd6f88cb77565283b257316267, Project source: https://github.com/bazelbuild/bazel.git
  metric          mean                median          stddev      pval   
    wall:  173.391s (+48.24%)   173.878s (+48.23%)     1.361s    0.90000 
     cpu:   81.277s ( +2.16%)    81.930s ( +1.80%)     2.404s    0.40000 
  system:   18.187s ( -5.11%)    18.450s ( -4.11%)     0.794s    0.40000 
  memory:   94.000s ( -0.70%)    94.000s ( -1.05%)     0.000s    0.40000 

2. cd3480e4d53160d20c634134d3917c4e53ac1c49 improved wall time by ~30%.

Bazel commit: 1517286783762647b4922ed1c656262002abedf1, Project commit: 1517286783762647b4922ed1c656262002abedf1, Project source: https://github.com/bazelbuild/bazel.git
  metric          mean                median          stddev      pval   
    wall:  184.560s             185.592s              1.563s             
     cpu:   82.817s              84.350s              2.996s             
  system:   19.877s              19.230s              1.111s             
  memory:   95.000MB             95.000MB             0.000MB            

Bazel commit: cd3480e4d53160d20c634134d3917c4e53ac1c49, Project commit: 1517286783762647b4922ed1c656262002abedf1, Project source: https://github.com/bazelbuild/bazel.git
  metric          mean                median          stddev      pval   
    wall:  128.454s  (-30.40%)  129.437s  (-30.26%)   1.443s     0.90000 
     cpu:   86.013s  ( +3.86%)   87.650s  ( +3.91%)   2.625s     0.40000 
  system:   20.917s  ( +5.23%)   20.820s  ( +8.27%)   0.227s     0.40000 
  memory:   94.000MB ( -1.05%)   94.000MB ( -1.05%)   0.000MB    0.90000 

Feature requests: what underlying problem are you trying to solve with this feature?

Maybe we can use this to figure out what in java_tools regressed the performance of Bazel and what corrected it (partially), and whether there's something more we can do.

What operating system are you running Bazel on?

Ubuntu18.04

Any other information, logs, or outputs that you want to share?

To reproduce, clone bazelbuild/bazel-bench and run:

# 1st benchmark.
bazel run benchmark --  \
--bazel_source=https://github.com/bazelbuild/bazel.git \
--project_source=https://github.com/bazelbuild/bazel.git \
--collect_memory \
--runs=3 \
--bazel_commits=5c92b11dc5e1decf1b2a30b72d9bdaaf6e3e4380,c282526c071389cd6f88cb77565283b257316267 \
--project_commits=c282526c071389cd6f88cb77565283b257316267 \
-- build //src:bazel

# 2nd benchmark
bazel run benchmark --  \
--bazel_source=https://github.com/bazelbuild/bazel.git \
--project_source=https://github.com/bazelbuild/bazel.git \
--collect_memory \
--data_directory=/tmp/.bazel-bench/out \
--runs=3 \
--bazel_commits=1517286783762647b4922ed1c656262002abedf1,cd3480e4d53160d20c634134d3917c4e53ac1c49 \
--project_commits=1517286783762647b4922ed1c656262002abedf1 \
-- build //src:bazel

Also observing the same issue. The workaround with the codestyle.xml is nice, but the formatting can be wildly different. Especially regarding annotations (on single line, annotation indentation, ...) so the workaround is in the end not really sufficient as you still have to always apply the style either manually or via a maven plugin.

The test //src/test/shell/bazel:bazel_bootstrap_distfile_test has started to fail in our postsubmit pipeline on Windows only.

Here's a list of broken jobs:

• https://buildkite.com/bazel/bazel-bazel/builds/14736#745cad19-516a-479b-8084-9154bf14c893
• https://buildkite.com/bazel/bazel-bazel/builds/14740#75c2bc28-cf58-4f5d-a152-a94ef3d5cde1
• https://buildkite.com/bazel/bazel-bazel/builds/14742#65cec04d-80c2-4661-8f61-e1c4276ddde3
• https://buildkite.com/bazel/bazel-bazel/builds/14743#a2981e51-fe4e-43da-bd32-efacfaf4fd5b

I have not seen it happen before job 14736, so this might be the culprit. 🤔 I'm not aware of any CI infrastructure or Windows image changes in the last days.

Here's an example log: https://storage.googleapis.com/bazel-untrusted-buildkite-artifacts/745cad19-516a-479b-8084-9154bf14c893/src%5Ctest%5Cshell%5Cbazel%5Cbazel_bootstrap_distfile_test%5Cattempt_1.log

The relevant error message seems to be:

ERROR: Analysis of target '//src:bazel_nojdk.exe' failed; build aborted: invalid registered execution platform '//:default_host_platform': no such target '//:default_host_platform': target 'default_host_platform' not declared in package '' defined by C:/b/iwgf237n/execroot/io_bazel/_tmp/a6b4c571d5cd258ae95c5906c6c25060/bazelbootstrap.iIshY86j/BUILD

When the test fails during a job, it will consistently fail during all attempts, always with the same error message. It is thus not a typically flaky test which fails with a random chance.

This builds on top of the commit "Separate java_tools into platform independent and prebuilt part.".

• java_tools release 11.0 is used
• java_tools platform independent repo is added
• java toolchain definitions appropriate for new release are added to tools/jdk/BUILD
• remote_java_tools are replaced with aliases
• tests are reenabled

Create a way to release a java_tools version (in addition to the linux, darwin and winodws versions) without including pre-built ijar and singlejar.

Macro default_java_toolchain used to create a java_toolchain rule with each attribute set using a platform dependent select. For example ijar attribute was set to //tools/jdk:ijar, which was a macro remote_java_tools_filegroup. That macro contains a select choosing the right @remote_java_tools repository.

With Java toolchainisation this become a fragile mechanism (selects need to match the toolchain...), which would be hard to maintain.

In previous commits java_toolchain_default was implemented, which is part of each java_tools repository and does not use any selects.

For backward compatibility default_java_toolchain was reimplemented to invoke java_toolchain_default. Implementation is not perfect, because it load 3 java_tools repositories (linux,darwin,windows). Not to invoke this misfeature, some macros were moved into utils.bzl. The use of default_java_toolchain should be discouraged.

remote_java_tools_filegroup/import targets in //tools/jdk/BUILD that were previous used only by default_java_toolchain were removed.

default_java_toolchain targets in //tools/jdk/BUILD were replaced with select aliases. Those will be removed when Java toolchainsation is complete.

legacy_toolchain was removed as a fall-back - it was an awkard one - going from Java 11 to Java 8 language level.

Running into this years later...

Just to make you aware, #12574 is still under investigation but may make another cherry pick necessary.

Releasing java_tools at:

commit 4 no regression febf4438cd cushon Restore GenClass's --temp_dir flag once more

commit 5 38% regression in wall 5c8e584e88 steinman Move WorkRequestHandler from build jar to Worker package. This is part of the cleanup associated with enabling JSON workers.

commit 8 TODO a4251eab69 steinman Refactor WorkRequestHandler to be an interface, of which Proto is one implementation.

Presumably the benefit of running under windows is worth such a performance tradeoff.

Just curious here, but do you know what the performance tradeoff actually was for regular files vs hard links? On any platform?

It is the cost to create copies of all of the files specified as inputs - read-streamed-writes to the tune of N files with M size.

Naively I can't see why plain files would perform worse than hard links, but I assume you found there was a difference?

The problem was not in performance of the executions, it was in preparing for them - with substantial numbers of very-low-wall-time operations, we could see starvation in the worker execution stages, wasting their numerous cpus waiting to get content prepared for an exec. Hardlinks and directory symlinks are required to help keep the average input processing size reasonable compared to executions.

If execution time is drastically (many times) higher than the maximum cost for a set of actions to have their files copied and mkdir'd into place, then this doesn't apply. When granularity increases, if input set culling does not happen, this is likely to get substantially (i.e. nonlinearly) worse.

Not sure is it important there, but i came across this error/warning when run ibazel:

iBazel [3:29PM]: Querying for files to watch...
Loading: 0 packages loaded
ERROR: /home/dzintars/.cache/bazel/_bazel_dzintars/73eb78004b7f7dbbf7672ded04d50930/external/bazel_tools/tools/osx/BUILD:80:19: every rule of type xcode_config_alias implicitly depends upon the target '@bazel_tools//tools/objc:host_xcodes', but this target could not be found because of: no such target '@bazel_tools//tools/objc:host_xcodes': target 'host_xcodes' not declared in package 'tools/objc' defined by /home/dzintars/.cache/bazel/_bazel_dzintars/73eb78004b7f7dbbf7672ded04d50930/external/bazel_tools/tools/objc/BUILD.bazel
ERROR: /home/dzintars/.cache/bazel/_bazel_dzintars/73eb78004b7f7dbbf7672ded04d50930/external/bazel_tools/tools/osx/BUILD:80:19: every rule of type xcode_config_alias implicitly depends upon the target '@bazel_tools//tools/objc:host_xcodes', but this target could not be found because of: no such target '@bazel_tools//tools/objc:host_xcodes': target 'host_xcodes' not declared in package 'tools/objc' defined by /home/dzintars/.cache/bazel/_bazel_dzintars/73eb78004b7f7dbbf7672ded04d50930/external/bazel_tools/tools/objc/BUILD.bazel
ERROR: Evaluation of subquery "deps(set(//platform/web/shell/server))" failed (did you want to use --keep_going?): errors were encountered while computing transitive closure
Loading: 1 packages loaded
Loading: 1 packages loaded
iBazel [3:29PM]: Bazel query failed: exit status 7

So i am looking forward to resolve this "warning".

` val gson = Gson()

        data class TestModel(
            val id: Int,
            val description: String
        )

        val jsonString = gson.toJson(TestModel(1,"Test"))

        println(jsonString)`

why jsonString is null ? who knows ? my kotlin version is 1.4.10. my jdk version is 1.8

A friendly ping, Intellij plugins are still broken with Bazel@HEAD: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1790

While looking for a Concurrent Multi Hash Map, I found Multimaps.synchronizedMultimap() wrapper which seems to be what I'm looking for. However, the documentation gives an example which I'm not picturing a situation when this might be useful: https://github.com/google/guava/blob/4709fe4d50cf5249b9e5e40fb8917c610831c317/guava/src/com/google/common/collect/Multimaps.java#L566-L583

My take is that put operations will still be outside this synchronized block. Am I getting it wrong?

I went to look for valid usages in the test suite, but only found one that didn't help me much understanding it.

My use case is that I don't really care for non-deterministic behavior as long as it does not result in a fail operation (e.g., throw an Exception). Even if this might not be an issue, I am still curious about a scenario where this should be employed.

ATTENTION! Please read and follow:

• if this is a question about how to build / test / query / deploy using Bazel, or a discussion starter, send it to bazel-discuss@googlegroups.com
• if this is a bug or feature request, fill the form below as best as you can.

Description of the problem / feature request:

I'm using bazel 2.0.0 with VS Code bazel plugin. I have set BAZEL_VS env variable to C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools globally. However, when I build a target in VS Code (clicking on build target from right-click menu in BAZEL BUILD TARGETS sidebar), it failed to probe compiler with following error logs:

ERROR: An error occurred during the fetch of repository 'local_config_cc':
   Traceback (most recent call last):
        File "C:/users/yangxi/_bazel_yangxi/y7qzzhw6/external/bazel_tools/tools/cpp/cc_configure.bzl", line 120                configure_windows_toolchain(<1 more arguments>)
        File "C:/users/yangxi/_bazel_yangxi/y7qzzhw6/external/bazel_tools/tools/cpp/windows_cc_configure.bzl", 
line 690, in configure_windows_toolchain
                _get_msvc_vars(repository_ctx, <1 more arguments>)
        File "C:/users/yangxi/_bazel_yangxi/y7qzzhw6/external/bazel_tools/tools/cpp/windows_cc_configure.bzl", 
line 535, in _get_msvc_vars
                setup_vc_env_vars(<2 more arguments>)
        File "C:/users/yangxi/_bazel_yangxi/y7qzzhw6/external/bazel_tools/tools/cpp/windows_cc_configure.bzl", 
line 318, in setup_vc_env_vars
                _check_env_vars(env_map, cmd, <1 more arguments>)
        File "C:/users/yangxi/_bazel_yangxi/y7qzzhw6/external/bazel_tools/tools/cpp/windows_cc_configure.bzl", 
line 324, in _check_env_vars
                auto_configure_fail(<1 more arguments>)
        File "C:/users/yangxi/_bazel_yangxi/y7qzzhw6/external/bazel_tools/tools/cpp/lib_cc_configure.bzl", line 112, in auto_configure_fail
                fail(<1 more arguments>)

Auto-Configuration Error: Setting up VC environment variables failed, INCLUDE is not set by the following command:
    "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\\VCVARSALL.BAT" amd64

It seems the specified version of VS is not used. Instead it is still using VS 2015.

At current, I could only build targets inside Developer Command Prompt for VS 2017, which have set up many stuffs.

Feature requests: what underlying problem are you trying to solve with this feature?

Replace this line with your answer.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Replace this line with your answer.

What operating system are you running Bazel on?

Windows 10

What's the output of bazel info release?

release 2.0.0

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

Replace this line with your answer.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Replace this line with your answer.

Have you found anything relevant by searching the web?

Replace these lines with your answer.

Places to look:

• StackOverflow: http://stackoverflow.com/questions/tagged/bazel
• GitHub issues: https://github.com/bazelbuild/bazel/issues
• email threads on https://groups.google.com/forum/#!forum/bazel-discuss

Any other information, logs, or outputs that you want to share?

Replace these lines with your answer.

If the files are large, upload as attachment or provide link.

I tried it and I think it is fixed, I will close it.

Fix error for class creation in 2020.2

CodeStyleManager#scheduleReformatWhenSettingsComputed is a new method in 2020.2 which throws an UnsupportedOperationException by default. We need to add delegation for this method to DelegatingCodeStyleManager to avoid running into an error when users create new classes.