profile
viewpoint
Josh Wills jwills N/A San Francisco, CA http://twitter.com/josh_wills Gainfully unemployed data person

grpc/grpc-web 5197

gRPC for Web Clients

broadinstitute/gatk 968

Official code repository for GATK versions 4 and up

HopkinsIDD/COVIDScenarioPipeline 89

Public shared code for doing scenario forecasting and creating reports for various governmental entities.

jwills/avro-json 29

Utilities for converting to and from JSON from Avro records via Hadoop streaming or Hive.

jwills/dbt-duckdb 11

dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)

broadinstitute/gatk-dataflow 4

Development dataflow

jwills/crunch-demo 4

A demo application for getting started with Apache Crunch.

jwills/avroplay 3

Me messing around with some Avro stuff

jwills/datafu 3

Hadoop library for large-scale data processing

jwills/attribution 2

MapReduce job for creating multitouch attribution models.

Pull request review commentapache/beam

[BEAM-10961] Strict dependency checking for sdks/java/io/gcp

 import groovy.json.JsonOutput  plugins { id 'org.apache.beam.module' } applyJavaNature(-+  //enableStrictDependencies:true,

@ibzib thanks for the tip. the only violation I see is that runners:core-java was usedUndeclared. The only fix I see is to add that explicitly.

shehzaadn-vd

comment created time in a minute

Pull request review commentapache/beam

[BEAM-10961] Strict dependency checking for sdks/java/io/gcp

 dependencies {     exclude group: 'io.grpc', module: 'grpc-grpclb'   }   compile library.java.google_cloud_core-  compile(library.java.google_cloud_core_grpc) {-    exclude group: 'io.grpc', module: 'grpc-core' // Use Beam's version-  }   compile library.java.google_cloud_datastore_v1_proto_client   compile library.java.google_cloud_pubsublite   // GCP PubSub client is used in TestPubSub   compile library.java.google_cloud_pubsub   compile library.java.google_cloud_spanner   compile library.java.google_http_client   compile library.java.google_http_client_jackson2-  compile library.java.grpc_alts   compile library.java.grpc_auth-  compile library.java.grpc_core-  compile library.java.grpc_context-  compile library.java.grpc_grpclb+  testImplementation 'io.grpc:grpc-core:1.27.2'   compile library.java.grpc_netty-  compile library.java.grpc_netty_shaded-  compile library.java.grpc_stub   compile library.java.grpc_google_cloud_pubsub_v1-  compile library.java.grpc_google_cloud_pubsublite_v1   compile library.java.guava-  compile library.java.hamcrest_core   compile library.java.jackson_databind   compile library.java.joda_time   compile library.java.junit   compile library.java.netty_handler-  compile library.java.netty_tcnative_boringssl_static-  compile library.java.proto_google_cloud_bigquery_storage_v1beta1   compile library.java.proto_google_cloud_bigtable_v2   compile library.java.proto_google_cloud_datastore_v1   compile library.java.proto_google_cloud_pubsub_v1   compile library.java.proto_google_cloud_pubsublite_v1-  compile library.java.proto_google_cloud_spanner_admin_database_v1   compile library.java.proto_google_common_protos   compile library.java.protobuf_java   compile library.java.slf4j_api+  compile "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0"

OK. Sounds good. I won't bother with the TODOs, and I'll add hamcrest to BeamModulePlugin.

shehzaadn-vd

comment created time in 8 minutes

pull request commentapache/beam

[BEAM-11695] [cherry-pick] Combiner packing in Dataflow (#13763)

R: @chamikaramj cc: @tvalentyn

yifanmai

comment created time in 8 minutes

PR opened apache/beam

[BEAM-11695] [cherry-pick] Combiner packing in Dataflow (#13763)

Previously, combiner packing did not work in Dataflow because the optimizations were only applied to the proto_pipeline object and not the pipeline object. This fixes combiner packing, and also makes it disabled by default unless enabled using the pre_optimize option.

Cherry-pick of #13763.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status<br>Build Status Build Status Build Status<br>Build Status<br>Build Status Build Status
Python Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status<br>Build Status<br>Build Status <br>Build Status Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests

See CI.md for more information about GitHub Actions CI.

+118 -32

0 comment

4 changed files

pr created time in 9 minutes

Pull request review commentapache/beam

[BEAM-10961] Strict dependency checking for sdks/java/io/gcp

 dependencies {     exclude group: 'io.grpc', module: 'grpc-grpclb'   }   compile library.java.google_cloud_core-  compile(library.java.google_cloud_core_grpc) {-    exclude group: 'io.grpc', module: 'grpc-core' // Use Beam's version-  }   compile library.java.google_cloud_datastore_v1_proto_client   compile library.java.google_cloud_pubsublite   // GCP PubSub client is used in TestPubSub   compile library.java.google_cloud_pubsub   compile library.java.google_cloud_spanner   compile library.java.google_http_client   compile library.java.google_http_client_jackson2-  compile library.java.grpc_alts   compile library.java.grpc_auth-  compile library.java.grpc_core-  compile library.java.grpc_context-  compile library.java.grpc_grpclb+  testImplementation 'io.grpc:grpc-core:1.27.2'   compile library.java.grpc_netty-  compile library.java.grpc_netty_shaded-  compile library.java.grpc_stub   compile library.java.grpc_google_cloud_pubsub_v1-  compile library.java.grpc_google_cloud_pubsublite_v1   compile library.java.guava-  compile library.java.hamcrest_core   compile library.java.jackson_databind   compile library.java.joda_time   compile library.java.junit   compile library.java.netty_handler-  compile library.java.netty_tcnative_boringssl_static-  compile library.java.proto_google_cloud_bigquery_storage_v1beta1   compile library.java.proto_google_cloud_bigtable_v2   compile library.java.proto_google_cloud_datastore_v1   compile library.java.proto_google_cloud_pubsub_v1   compile library.java.proto_google_cloud_pubsublite_v1-  compile library.java.proto_google_cloud_spanner_admin_database_v1   compile library.java.proto_google_common_protos   compile library.java.protobuf_java   compile library.java.slf4j_api+  compile "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0"

the version coming in from google_cloud_platform_libraries_bom is unclear to me.

That's totally fine. The whole point of using the bom is that it provides a single source of truth for dependency versions. If the version provided by the bom mismatches some version the build is actually using, it's a problem.

org.hamcrest:hamcrest is not available in BeamModulePlugin

I've seen this one in enough other modules that it probably should be placed in BeamModulePlugin.

Skipping, but putting in a TODO in the code to match version numbers.

Try replacing these anyway, don't worry about matching the versions exactly. In most cases I expect that it shouldn't matter if we use a slightly newer or older version than one of the transitive dependencies, as long as the versions are compatible. (If we don't match versions, Gradle will do that for us anyway when it resolves dependencies.)

shehzaadn-vd

comment created time in 12 minutes

Pull request review commentapache/beam

[BEAM-10961] Strict dependency checking for sdks/java/io/gcp

 dependencies {     exclude group: 'io.grpc', module: 'grpc-grpclb'   }   compile library.java.google_cloud_core-  compile(library.java.google_cloud_core_grpc) {-    exclude group: 'io.grpc', module: 'grpc-core' // Use Beam's version-  }   compile library.java.google_cloud_datastore_v1_proto_client   compile library.java.google_cloud_pubsublite   // GCP PubSub client is used in TestPubSub   compile library.java.google_cloud_pubsub   compile library.java.google_cloud_spanner   compile library.java.google_http_client   compile library.java.google_http_client_jackson2-  compile library.java.grpc_alts   compile library.java.grpc_auth-  compile library.java.grpc_core-  compile library.java.grpc_context-  compile library.java.grpc_grpclb+  testImplementation 'io.grpc:grpc-core:1.27.2'   compile library.java.grpc_netty-  compile library.java.grpc_netty_shaded-  compile library.java.grpc_stub   compile library.java.grpc_google_cloud_pubsub_v1-  compile library.java.grpc_google_cloud_pubsublite_v1   compile library.java.guava-  compile library.java.hamcrest_core   compile library.java.jackson_databind   compile library.java.joda_time   compile library.java.junit   compile library.java.netty_handler-  compile library.java.netty_tcnative_boringssl_static-  compile library.java.proto_google_cloud_bigquery_storage_v1beta1   compile library.java.proto_google_cloud_bigtable_v2   compile library.java.proto_google_cloud_datastore_v1   compile library.java.proto_google_cloud_pubsub_v1   compile library.java.proto_google_cloud_pubsublite_v1-  compile library.java.proto_google_cloud_spanner_admin_database_v1   compile library.java.proto_google_common_protos   compile library.java.protobuf_java   compile library.java.slf4j_api+  compile "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0"

@ibzib Comparing the lines added here with what's available in BeamModulePlugin and my proposed plan to use what's available where possible:

  • com.google.api.grpc:proto-google-cloud-bigquerystorage-v1 is not available - as you pointed out.
  • com.google.api.grpc:proto-google-cloud-bigtable-admin-v2 is not available
  • com.google.api.grpc:proto-google-cloud-spanner-v1 is not available
  • com.google.api:api-common is available - but the version coming in from google_cloud_platform_libraries_bom is unclear to me. Replacing.
  • com.google.apis:google-api-services-storage is a match. Replacing.
  • com.google.code.gson:gson:2.8.6 is a match. Replacing.
  • com.google.errorprone:error_prone_annotations is 2.3.3 here, but 2.3.1 in BeamModulePlugin. Skipping, but putting in a TODO in the code to match version numbers.
  • com.google.protobuf:protobuf-java-util is 3.11.1 here, but 3.12.0 in BeamModulePlugin. Skipping, but putting in a TODO in the code to match version numbers.
  • io.grpc:grpc-api is available - but the version coming in from google_cloud_platform_libraries_bom is unclear to me. Replacing.
  • org.apache.beam:beam-vendor-guava-26_0-jre is available. ** Replacing**.
  • org.apache.httpcomponents:httpclient is 4.5.12 here, but 4.5.10 in BeamModulePlugin. Skipping but putting in a TODO in the code to match version numbers.
  • org.apache.httpcomponents:httpcore is 4.4.13 here, but 4.4.12 in BeamModulePlugin. ** Skipping** but putting in a TODO in the code to match version numbers.
  • org.hamcrest:hamcrest is not available in BeamModulePlugin
  • org.threethen:threetenbp is not available in BeamModulePlugin
  • compile "org.apache.beam:beam-vendor-grpc-1_26_0:0.3" is available. Replacing
shehzaadn-vd

comment created time in 22 minutes

pull request commentapache/beam

[BEAM-11648] In preparation for new BigQuery sink, update GCP BOM

The problem was that I had not yet pushed my client to github. after pushing, the error went away.

@suztomo I'm now getting the below errors. I'm not entirely sure how to interpret the output of this tool. Most of the errors seem related to grpc. Does this mean that some other component is pulling in a conflicting version of grpc?

com.google.cloud.tools.opensource.classpath.LinkageCheckResultException: Found 14 linkage errors io.grpc.internal.ManagedChannelImpl (io.grpc:grpc-core:1.35.0) at com.google.cloud.tools.opensource.classpath.LinkageCheckerMain.main(LinkageCheckerMain.java:76) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.InternalConfigSelector$Result's method getInterceptor() but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method createResolvingOobChannelBuilder(String, io.grpc.ChannelCredentials) is not found; referenced by 1 class file io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method createResolvingOobChannelBuilder(String, io.grpc.ChannelCredentials) but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method getChannelCredentials() is not found; referenced by 1 class file io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method getChannelCredentials() but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method getUnsafeChannelCredentials() is not found; referenced by 1 class file io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method getUnsafeChannelCredentials() but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.PooledByteBufAllocator's method <init>(boolean, int, int, int, int, int, int, boolean) is not found; referenced by 1 class file io.grpc.netty.Utils (io.grpc:grpc-netty:1.35.0) Cause: Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.PooledByteBufAllocator's method <init>(boolean, int, int, int, int, int, int, boolean) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() is not found; referenced by 2 class files com.google.cloud.bigquery.Acl (com.google.cloud:google-cloud-bigquery:1.126.3) com.google.cloud.bigquery.DatasetInfo (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.Dataset$Access's method setRoutine(com.google.api.services.bigquery.model.RoutineReference) is not found; referenced by 1 class file com.google.cloud.bigquery.Acl (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method setRoutine(com.google.api.services.bigquery.model.RoutineReference) but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.HivePartitioningOptions's method setRequirePartitionFilter(Boolean) is not found; referenced by 1 class file com.google.cloud.bigquery.HivePartitioningOptions (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.HivePartitioningOptions's method setRequirePartitionFilter(Boolean) but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.HivePartitioningOptions's method getRequirePartitionFilter() is not found; referenced by 1 class file com.google.cloud.bigquery.HivePartitioningOptions (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.HivePartitioningOptions's method getRequirePartitionFilter() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) is not found; referenced by 2 class files io.netty.handler.codec.http2.DefaultHttp2DataFrame (io.netty:netty-codec-http2:4.1.52.Final) io.netty.handler.codec.spdy.DefaultSpdyDataFrame (io.netty:netty-codec-http:4.1.52.Final) Cause: Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) (io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) is not found; referenced by 2 class files io.netty.handler.codec.http.DefaultFullHttpResponse (io.netty:netty-codec-http:4.1.52.Final) io.netty.handler.codec.http.DefaultFullHttpRequest (io.netty:netty-codec-http:4.1.52.Final) Cause: Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) Problematic artifacts in the dependency tree: io.grpc:grpc-api:1.34.1 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) and 2 other dependency paths. io.grpc:grpc-core:1.35.0 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) and 5 other dependency paths. io.netty:netty-buffer:4.1.51.Final is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) and 3 other dependency paths. io.grpc:grpc-netty:1.35.0 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) com.google.cloud:google-cloud-bigquery:1.126.3 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) io.netty:netty-codec-http2:4.1.52.Final is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) io.netty:netty-codec-http:4.1.52.Final is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile)

reuvenlax

comment created time in 30 minutes

pull request commentapache/beam

[BEAM-11695] Combiner packing in Dataflow

I think we will want to cherry-pick this.

yifanmai

comment created time in 31 minutes

pull request commentapache/beam

[BEAM-11415] Enabling new ValidatesRunner framework on Jenkins. [DO NOT MERGE]

Run Go PostCommit

youngoli

comment created time in 32 minutes

PR opened apache/beam

[BEAM-11689] (release-2.28.0) Add public.nexus.pentaho.org to offline repositories.

cherry-pick #13823

R: @chamikaramj


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status<br>Build Status Build Status Build Status<br>Build Status<br>Build Status Build Status
Python Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status<br>Build Status<br>Build Status <br>Build Status Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests

See CI.md for more information about GitHub Actions CI.

+1 -0

0 comment

1 changed file

pr created time in an hour

pull request commentapache/beam

[BEAM-11705] Fixing ignore_insert_id implementation

Run Python 3.8 PostCommit

pabloem

comment created time in an hour

pull request commentapache/beam

[BEAM-11408] Integrate BigQuery sink streaming inserts with GroupIntoBatches

Run Java PostCommit

nehsyc

comment created time in an hour

PR opened apache/beam

[BEAM-11705] Fixing ignore_insert_id implementation

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status<br>Build Status Build Status Build Status<br>Build Status<br>Build Status Build Status
Python Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status<br>Build Status<br>Build Status <br>Build Status Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests

See CI.md for more information about GitHub Actions CI.

+2 -19

0 comment

2 changed files

pr created time in an hour

pull request commentapache/beam

Cherrypick Merge pull request #13797 from [BEAM-11679] Override PubsubUnboundedS…

R: @chamikaramj

y1chi

comment created time in an hour

PR opened apache/beam

Cherrypick Merge pull request #13797 from [BEAM-11679] Override PubsubUnboundedS…

…ource transform for dataflow runner v2

[BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status<br>Build Status Build Status Build Status<br>Build Status<br>Build Status Build Status
Python Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status<br>Build Status<br>Build Status <br>Build Status Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests

See CI.md for more information about GitHub Actions CI.

+185 -65

0 comment

7 changed files

pr created time in an hour

pull request commentapache/beam

[BEAM-11695] Combiner packing in Dataflow

This was merged after release branch was cut. If this is a blocker for 2.28.0 please send in a cherry-pick to the release branch: https://github.com/apache/beam/tree/release-2.28.0

yifanmai

comment created time in an hour

Pull request review commentapache/beam

[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner

 static void verifyDoFnSupportedStreaming(DoFn<?, ?> fn) {   }    static void verifyDoFnSupported(DoFn<?, ?> fn, boolean streaming) {-    if (DoFnSignatures.usesSetState(fn)) {

Thanks for the support status confirmation! I would image we still want to keep this checking part for runner v2 as well.

reuvenlax

comment created time in an hour

pull request commentapache/beam

[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner

Ah - good point. We still need to exclude those tests for the V2 runner.

On Wed, Jan 27, 2021 at 4:03 PM Reuven Lax relax@google.com wrote:

No - this PR adds support to the legacy runner only.

On Wed, Jan 27, 2021 at 3:52 PM Boyuan Zhang notifications@github.com wrote:

@boyuanzz commented on this pull request.

In runners/google-cloud-dataflow-java/build.gradle https://github.com/apache/beam/pull/13802#discussion_r565721856:

@@ -165,8 +163,6 @@ def commonLegacyExcludeCategories = [ def commonRunnerV2ExcludeCategories = [ 'org.apache.beam.sdk.testing.UsesCommittedMetrics', 'org.apache.beam.sdk.testing.UsesGaugeMetrics',

  •    'org.apache.beam.sdk.testing.UsesSetState',
    

Are MapState and SetState also supported in dataflow runner v2(unified worker) within this changes?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/apache/beam/pull/13802#pullrequestreview-577873503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAYJVNZAEBYVMPTRQAKZGDS4CRKZANCNFSM4WP3BTDA .

reuvenlax

comment created time in an hour

pull request commentapache/beam

[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner

No - this PR adds support to the legacy runner only.

On Wed, Jan 27, 2021 at 3:52 PM Boyuan Zhang notifications@github.com wrote:

@boyuanzz commented on this pull request.

In runners/google-cloud-dataflow-java/build.gradle https://github.com/apache/beam/pull/13802#discussion_r565721856:

@@ -165,8 +163,6 @@ def commonLegacyExcludeCategories = [ def commonRunnerV2ExcludeCategories = [ 'org.apache.beam.sdk.testing.UsesCommittedMetrics', 'org.apache.beam.sdk.testing.UsesGaugeMetrics',

  •    'org.apache.beam.sdk.testing.UsesSetState',
    

Are MapState and SetState also supported in dataflow runner v2(unified worker) within this changes?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/apache/beam/pull/13802#pullrequestreview-577873503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAYJVNZAEBYVMPTRQAKZGDS4CRKZANCNFSM4WP3BTDA .

reuvenlax

comment created time in an hour

pull request commentapache/beam

[BEAM-11689] Add public.nexus.pentaho.org to offline repositories.

Can you send a cherry-pick to 2.28.0 branch as well ?

ibzib

comment created time in an hour

pull request commentapache/beam

[BEAM-11643] Add BigDecimal support for SpannerIO

2.28.0 release branch was already cut: https://github.com/apache/beam/tree/release-2.28.0 So assuming this is a blocker, can you propose a cherry-pick to that branch as well ?

nielm

comment created time in an hour

push eventapache/beam

Yichi Zhang

commit sha a9d861273bc1c64a984c73209504461bd63fb4be

[BEAM-11679] Override PubsubUnboundedSource transform for dataflow runner v2

view details

Yichi Zhang

commit sha 56cebdacc8287ddd804d4e5abdbbcd54e6d4d597

Populate PubsubMessage message id

view details

Yichi Zhang

commit sha 02ec09c04f33cd2584db88a004ff26ac1840f771

Enable tests

view details

Yichi Zhang

commit sha 197d33802fb4eab3d1829ba122bfe1b43e16949f

Exclude failing FhirIO ITs

view details

Yichi Zhang

commit sha bc7930e60f6a8206273a3a8373e25d03085d3e61

Include runner v2 IT tests in java post commit

view details

Boyuan Zhang

commit sha ebc3bc31e1f4185241e4a2cf2212772fd7c1fb66

Merge pull request #13797 from [BEAM-11679] Override PubsubUnboundedSource transform for dataflow runner v2 [BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…

view details

push time in an hour

PR merged apache/beam

[BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…

…nner v2

And also include RunnerV2 integration tests in post commit.

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status<br>Build Status Build Status Build Status<br>Build Status<br>Build Status Build Status
Python Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status<br>Build Status<br>Build Status <br>Build Status Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests

See CI.md for more information about GitHub Actions CI.

+185 -65

6 comments

7 changed files

y1chi

pr closed time in an hour

pull request commentapache/beam

[BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…

Can you also propose a cherry-pick to 2.28.0 release branch ? (assuming this is a blocker)

y1chi

comment created time in an hour

push eventapache/beam

nielm

commit sha 8070497559bafb5717c9f3d43e4d662a72d7d919

Add BigDecimal support for SpannerIO Since Numeric types were added to Spanner, it is possible to use BigDecimal values in Spanner mutations.

view details

Chamikara Jayalath

commit sha 9a4505e8de1b7e30e19a31eee1a18ca6e744351b

Merge pull request #13762: [BEAM-11643] Add BigDecimal support for SpannerIO

view details

push time in an hour

PR merged apache/beam

[BEAM-11643] Add BigDecimal support for SpannerIO

Since Numeric types were added to Spanner, it is possible to use BigDecimal values in Spanner mutations.

This PR adds this support in the MutationSizeEstimator.

Support for writing these types was already added in #12818


Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status<br>Build Status Build Status Build Status<br>Build Status<br>Build Status Build Status
Python Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status<br>Build Status<br>Build Status <br>Build Status Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests

See CI.md for more information about GitHub Actions CI.

+48 -0

7 comments

2 changed files

nielm

pr closed time in an hour

Pull request review commentapache/beam

[BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…

 task javaPreCommitPortabilityApi() {  task javaPostCommit() {   dependsOn ":runners:google-cloud-dataflow-java:postCommit"+  dependsOn ":runners:google-cloud-dataflow-java:postCommitRunnerV2"

I like the idea that we add runner v2 post commit into current test suite. One further question I have is that should we consider have dataflow post commit separated from java post commit? cc: @tysonjh

y1chi

comment created time in an hour

Pull request review commentapache/beam

[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner

 def commonLegacyExcludeCategories = [ def commonRunnerV2ExcludeCategories = [         'org.apache.beam.sdk.testing.UsesCommittedMetrics',         'org.apache.beam.sdk.testing.UsesGaugeMetrics',-        'org.apache.beam.sdk.testing.UsesSetState',

Are MapState and SetState also supported in dataflow runner v2(unified worker) within this changes?

reuvenlax

comment created time in an hour

PR closed apache/beam

[BEAM-11613] Updates Dataflow pipelines to always use SDK Harness containers hosted in GCR python

This basically results multi-language pipelines using GCR images for external SDKs.

This update only works for released Beam versions. For unreleased Beam code SDK harness containers for external SDKs have to be manually overridden similar to today.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status<br>Build Status Build Status Build Status<br>Build Status<br>Build Status Build Status
Python Build Status<br>Build Status<br>Build Status Build Status<br>Build Status<br>Build Status Build Status<br>Build Status --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status<br>Build Status<br>Build Status <br>Build Status Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests

See CI.md for more information about GitHub Actions CI.

+57 -12

3 comments

2 changed files

chamikaramj

pr closed time in 2 hours

pull request commentapache/beam

[BEAM-11613] Updates Dataflow pipelines to always use SDK Harness containers hosted in GCR

This PR is on hold till we finalize some other things in the Dataflow side. Will close for now and reopen when ready.

chamikaramj

comment created time in 2 hours

more