grpc/grpc-web 5197
gRPC for Web Clients
Official code repository for GATK versions 4 and up
HopkinsIDD/COVIDScenarioPipeline 89
Public shared code for doing scenario forecasting and creating reports for various governmental entities.
Utilities for converting to and from JSON from Avro records via Hadoop streaming or Hive.
dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)
broadinstitute/gatk-dataflow 4
Development dataflow
A demo application for getting started with Apache Crunch.
Me messing around with some Avro stuff
Hadoop library for large-scale data processing
MapReduce job for creating multitouch attribution models.
Pull request review commentapache/beam
[BEAM-10961] Strict dependency checking for sdks/java/io/gcp
import groovy.json.JsonOutput plugins { id 'org.apache.beam.module' } applyJavaNature(-+ //enableStrictDependencies:true,
@ibzib thanks for the tip. the only violation I see is that runners:core-java was usedUndeclared. The only fix I see is to add that explicitly.
comment created time in a minute
Pull request review commentapache/beam
[BEAM-10961] Strict dependency checking for sdks/java/io/gcp
dependencies { exclude group: 'io.grpc', module: 'grpc-grpclb' } compile library.java.google_cloud_core- compile(library.java.google_cloud_core_grpc) {- exclude group: 'io.grpc', module: 'grpc-core' // Use Beam's version- } compile library.java.google_cloud_datastore_v1_proto_client compile library.java.google_cloud_pubsublite // GCP PubSub client is used in TestPubSub compile library.java.google_cloud_pubsub compile library.java.google_cloud_spanner compile library.java.google_http_client compile library.java.google_http_client_jackson2- compile library.java.grpc_alts compile library.java.grpc_auth- compile library.java.grpc_core- compile library.java.grpc_context- compile library.java.grpc_grpclb+ testImplementation 'io.grpc:grpc-core:1.27.2' compile library.java.grpc_netty- compile library.java.grpc_netty_shaded- compile library.java.grpc_stub compile library.java.grpc_google_cloud_pubsub_v1- compile library.java.grpc_google_cloud_pubsublite_v1 compile library.java.guava- compile library.java.hamcrest_core compile library.java.jackson_databind compile library.java.joda_time compile library.java.junit compile library.java.netty_handler- compile library.java.netty_tcnative_boringssl_static- compile library.java.proto_google_cloud_bigquery_storage_v1beta1 compile library.java.proto_google_cloud_bigtable_v2 compile library.java.proto_google_cloud_datastore_v1 compile library.java.proto_google_cloud_pubsub_v1 compile library.java.proto_google_cloud_pubsublite_v1- compile library.java.proto_google_cloud_spanner_admin_database_v1 compile library.java.proto_google_common_protos compile library.java.protobuf_java compile library.java.slf4j_api+ compile "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0"
OK. Sounds good. I won't bother with the TODOs, and I'll add hamcrest to BeamModulePlugin.
comment created time in 8 minutes
pull request commentapache/beam
[BEAM-11695] [cherry-pick] Combiner packing in Dataflow (#13763)
R: @chamikaramj cc: @tvalentyn
comment created time in 8 minutes
PR opened apache/beam
Previously, combiner packing did not work in Dataflow because the optimizations were only applied to the proto_pipeline object and not the pipeline object. This fixes combiner packing, and also makes it disabled by default unless enabled using the pre_optimize option.
Cherry-pick of #13763.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Choose reviewer(s) and mention them in a comment (
R: @username). - [ ] Format the pull request title like
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
| Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 |
|---|---|---|---|---|---|---|
| Go | --- | --- | --- | |||
| Java | ||||||
| Python | --- | --- | ||||
| XLang | --- | --- |
Pre-Commit Tests Status (on master branch)
| --- | Java | Python | Go | Website | Whitespace | Typescript |
|---|---|---|---|---|---|---|
| Non-portable | ||||||
| Portable | --- | --- | --- | --- | --- |
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
pr created time in 9 minutes
Pull request review commentapache/beam
[BEAM-10961] Strict dependency checking for sdks/java/io/gcp
dependencies { exclude group: 'io.grpc', module: 'grpc-grpclb' } compile library.java.google_cloud_core- compile(library.java.google_cloud_core_grpc) {- exclude group: 'io.grpc', module: 'grpc-core' // Use Beam's version- } compile library.java.google_cloud_datastore_v1_proto_client compile library.java.google_cloud_pubsublite // GCP PubSub client is used in TestPubSub compile library.java.google_cloud_pubsub compile library.java.google_cloud_spanner compile library.java.google_http_client compile library.java.google_http_client_jackson2- compile library.java.grpc_alts compile library.java.grpc_auth- compile library.java.grpc_core- compile library.java.grpc_context- compile library.java.grpc_grpclb+ testImplementation 'io.grpc:grpc-core:1.27.2' compile library.java.grpc_netty- compile library.java.grpc_netty_shaded- compile library.java.grpc_stub compile library.java.grpc_google_cloud_pubsub_v1- compile library.java.grpc_google_cloud_pubsublite_v1 compile library.java.guava- compile library.java.hamcrest_core compile library.java.jackson_databind compile library.java.joda_time compile library.java.junit compile library.java.netty_handler- compile library.java.netty_tcnative_boringssl_static- compile library.java.proto_google_cloud_bigquery_storage_v1beta1 compile library.java.proto_google_cloud_bigtable_v2 compile library.java.proto_google_cloud_datastore_v1 compile library.java.proto_google_cloud_pubsub_v1 compile library.java.proto_google_cloud_pubsublite_v1- compile library.java.proto_google_cloud_spanner_admin_database_v1 compile library.java.proto_google_common_protos compile library.java.protobuf_java compile library.java.slf4j_api+ compile "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0"
the version coming in from google_cloud_platform_libraries_bom is unclear to me.
That's totally fine. The whole point of using the bom is that it provides a single source of truth for dependency versions. If the version provided by the bom mismatches some version the build is actually using, it's a problem.
org.hamcrest:hamcrest is not available in BeamModulePlugin
I've seen this one in enough other modules that it probably should be placed in BeamModulePlugin.
Skipping, but putting in a TODO in the code to match version numbers.
Try replacing these anyway, don't worry about matching the versions exactly. In most cases I expect that it shouldn't matter if we use a slightly newer or older version than one of the transitive dependencies, as long as the versions are compatible. (If we don't match versions, Gradle will do that for us anyway when it resolves dependencies.)
comment created time in 12 minutes
Pull request review commentapache/beam
[BEAM-10961] Strict dependency checking for sdks/java/io/gcp
dependencies { exclude group: 'io.grpc', module: 'grpc-grpclb' } compile library.java.google_cloud_core- compile(library.java.google_cloud_core_grpc) {- exclude group: 'io.grpc', module: 'grpc-core' // Use Beam's version- } compile library.java.google_cloud_datastore_v1_proto_client compile library.java.google_cloud_pubsublite // GCP PubSub client is used in TestPubSub compile library.java.google_cloud_pubsub compile library.java.google_cloud_spanner compile library.java.google_http_client compile library.java.google_http_client_jackson2- compile library.java.grpc_alts compile library.java.grpc_auth- compile library.java.grpc_core- compile library.java.grpc_context- compile library.java.grpc_grpclb+ testImplementation 'io.grpc:grpc-core:1.27.2' compile library.java.grpc_netty- compile library.java.grpc_netty_shaded- compile library.java.grpc_stub compile library.java.grpc_google_cloud_pubsub_v1- compile library.java.grpc_google_cloud_pubsublite_v1 compile library.java.guava- compile library.java.hamcrest_core compile library.java.jackson_databind compile library.java.joda_time compile library.java.junit compile library.java.netty_handler- compile library.java.netty_tcnative_boringssl_static- compile library.java.proto_google_cloud_bigquery_storage_v1beta1 compile library.java.proto_google_cloud_bigtable_v2 compile library.java.proto_google_cloud_datastore_v1 compile library.java.proto_google_cloud_pubsub_v1 compile library.java.proto_google_cloud_pubsublite_v1- compile library.java.proto_google_cloud_spanner_admin_database_v1 compile library.java.proto_google_common_protos compile library.java.protobuf_java compile library.java.slf4j_api+ compile "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0"
@ibzib Comparing the lines added here with what's available in BeamModulePlugin and my proposed plan to use what's available where possible:
- com.google.api.grpc:proto-google-cloud-bigquerystorage-v1 is not available - as you pointed out.
- com.google.api.grpc:proto-google-cloud-bigtable-admin-v2 is not available
- com.google.api.grpc:proto-google-cloud-spanner-v1 is not available
- com.google.api:api-common is available - but the version coming in from google_cloud_platform_libraries_bom is unclear to me. Replacing.
- com.google.apis:google-api-services-storage is a match. Replacing.
- com.google.code.gson:gson:2.8.6 is a match. Replacing.
- com.google.errorprone:error_prone_annotations is 2.3.3 here, but 2.3.1 in BeamModulePlugin. Skipping, but putting in a TODO in the code to match version numbers.
- com.google.protobuf:protobuf-java-util is 3.11.1 here, but 3.12.0 in BeamModulePlugin. Skipping, but putting in a TODO in the code to match version numbers.
- io.grpc:grpc-api is available - but the version coming in from google_cloud_platform_libraries_bom is unclear to me. Replacing.
- org.apache.beam:beam-vendor-guava-26_0-jre is available. ** Replacing**.
- org.apache.httpcomponents:httpclient is 4.5.12 here, but 4.5.10 in BeamModulePlugin. Skipping but putting in a TODO in the code to match version numbers.
- org.apache.httpcomponents:httpcore is 4.4.13 here, but 4.4.12 in BeamModulePlugin. ** Skipping** but putting in a TODO in the code to match version numbers.
- org.hamcrest:hamcrest is not available in BeamModulePlugin
- org.threethen:threetenbp is not available in BeamModulePlugin
- compile "org.apache.beam:beam-vendor-grpc-1_26_0:0.3" is available. Replacing
comment created time in 22 minutes
pull request commentapache/beam
[BEAM-11648] In preparation for new BigQuery sink, update GCP BOM
The problem was that I had not yet pushed my client to github. after pushing, the error went away.
@suztomo I'm now getting the below errors. I'm not entirely sure how to interpret the output of this tool. Most of the errors seem related to grpc. Does this mean that some other component is pulling in a conflicting version of grpc?
com.google.cloud.tools.opensource.classpath.LinkageCheckResultException: Found 14 linkage errors io.grpc.internal.ManagedChannelImpl (io.grpc:grpc-core:1.35.0) at com.google.cloud.tools.opensource.classpath.LinkageCheckerMain.main(LinkageCheckerMain.java:76) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.InternalConfigSelector$Result's method getInterceptor() but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method createResolvingOobChannelBuilder(String, io.grpc.ChannelCredentials) is not found; referenced by 1 class file io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method createResolvingOobChannelBuilder(String, io.grpc.ChannelCredentials) but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method getChannelCredentials() is not found; referenced by 1 class file io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method getChannelCredentials() but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.grpc:grpc-api:1.34.1) io.grpc.LoadBalancer$Helper's method getUnsafeChannelCredentials() is not found; referenced by 1 class file io.grpc.util.ForwardingLoadBalancerHelper (io.grpc:grpc-core:1.35.0) Cause: Dependency conflict: io.grpc:grpc-api:1.34.1 does not define io.grpc.LoadBalancer$Helper's method getUnsafeChannelCredentials() but io.grpc:grpc-api:1.35.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) / io.grpc:grpc-api:1.35.0 (compile) (io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.PooledByteBufAllocator's method <init>(boolean, int, int, int, int, int, int, boolean) is not found; referenced by 1 class file io.grpc.netty.Utils (io.grpc:grpc-netty:1.35.0) Cause: Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.PooledByteBufAllocator's method <init>(boolean, int, int, int, int, int, int, boolean) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() is not found; referenced by 2 class files com.google.cloud.bigquery.Acl (com.google.cloud:google-cloud-bigquery:1.126.3) com.google.cloud.bigquery.DatasetInfo (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method getRoutine() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.Dataset$Access's method setRoutine(com.google.api.services.bigquery.model.RoutineReference) is not found; referenced by 1 class file com.google.cloud.bigquery.Acl (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.Dataset$Access's method setRoutine(com.google.api.services.bigquery.model.RoutineReference) but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.HivePartitioningOptions's method setRequirePartitionFilter(Boolean) is not found; referenced by 1 class file com.google.cloud.bigquery.HivePartitioningOptions (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.HivePartitioningOptions's method setRequirePartitionFilter(Boolean) but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10) com.google.api.services.bigquery.model.HivePartitioningOptions's method getRequirePartitionFilter() is not found; referenced by 1 class file com.google.cloud.bigquery.HivePartitioningOptions (com.google.cloud:google-cloud-bigquery:1.126.3) Cause: Dependency conflict: com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 does not define com.google.api.services.bigquery.model.HivePartitioningOptions's method getRequirePartitionFilter() but com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) / com.google.apis:google-api-services-bigquery:v2-rev20201030-1.31.0 (compile) (io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) is not found; referenced by 2 class files io.netty.handler.codec.http2.DefaultHttp2DataFrame (io.netty:netty-codec-http2:4.1.52.Final) io.netty.handler.codec.spdy.DefaultSpdyDataFrame (io.netty:netty-codec-http:4.1.52.Final) Cause: Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method ensureAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) (io.netty:netty-buffer:4.1.51.Final) io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) is not found; referenced by 2 class files io.netty.handler.codec.http.DefaultFullHttpResponse (io.netty:netty-codec-http:4.1.52.Final) io.netty.handler.codec.http.DefaultFullHttpRequest (io.netty:netty-codec-http:4.1.52.Final) Cause: Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) Dependency conflict: io.netty:netty-buffer:4.1.51.Final does not define io.netty.buffer.ByteBufUtil's method isAccessible(io.netty.buffer.ByteBuf) but io.netty:netty-buffer:4.1.52.Final defines it. selected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) unselected: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile) / io.netty:netty-buffer:4.1.52.Final (compile) Problematic artifacts in the dependency tree: io.grpc:grpc-api:1.34.1 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / io.grpc:grpc-api:1.34.1 (compile) and 2 other dependency paths. io.grpc:grpc-core:1.35.0 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-core:1.35.0 (compile) and 5 other dependency paths. io.netty:netty-buffer:4.1.51.Final is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.netty:netty-handler:4.1.51.Final (compile) / io.netty:netty-buffer:4.1.51.Final (compile) and 3 other dependency paths. io.grpc:grpc-netty:1.35.0 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.apis:google-api-services-bigquery:v2-rev20200719-1.30.10 (compile) com.google.cloud:google-cloud-bigquery:1.126.3 is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / com.google.cloud:google-cloud-bigquerystorage:1.8.0 (compile) / com.google.cloud:google-cloud-bigquery:1.126.3 (compile) io.netty:netty-codec-http2:4.1.52.Final is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) io.netty:netty-codec-http:4.1.52.Final is at: org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.29.0-SNAPSHOT / org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.29.0-SNAPSHOT (compile) / io.grpc:grpc-netty:1.35.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-codec-http:4.1.52.Final (compile)
comment created time in 30 minutes
pull request commentapache/beam
[BEAM-11695] Combiner packing in Dataflow
I think we will want to cherry-pick this.
comment created time in 31 minutes
pull request commentapache/beam
[BEAM-11415] Enabling new ValidatesRunner framework on Jenkins. [DO NOT MERGE]
Run Go PostCommit
comment created time in 32 minutes
PR opened apache/beam
cherry-pick #13823
R: @chamikaramj
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Choose reviewer(s) and mention them in a comment (
R: @username). - [ ] Format the pull request title like
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
| Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 |
|---|---|---|---|---|---|---|
| Go | --- | --- | --- | |||
| Java | ||||||
| Python | --- | --- | ||||
| XLang | --- | --- |
Pre-Commit Tests Status (on master branch)
| --- | Java | Python | Go | Website | Whitespace | Typescript |
|---|---|---|---|---|---|---|
| Non-portable | ||||||
| Portable | --- | --- | --- | --- | --- |
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
pr created time in an hour
pull request commentapache/beam
[BEAM-11705] Fixing ignore_insert_id implementation
Run Python 3.8 PostCommit
comment created time in an hour
pull request commentapache/beam
[BEAM-11408] Integrate BigQuery sink streaming inserts with GroupIntoBatches
Run Java PostCommit
comment created time in an hour
PR opened apache/beam
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Choose reviewer(s) and mention them in a comment (
R: @username). - [ ] Format the pull request title like
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
| Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 |
|---|---|---|---|---|---|---|
| Go | --- | --- | --- | |||
| Java | ||||||
| Python | --- | --- | ||||
| XLang | --- | --- |
Pre-Commit Tests Status (on master branch)
| --- | Java | Python | Go | Website | Whitespace | Typescript |
|---|---|---|---|---|---|---|
| Non-portable | ||||||
| Portable | --- | --- | --- | --- | --- |
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
pr created time in an hour
pull request commentapache/beam
Cherrypick Merge pull request #13797 from [BEAM-11679] Override PubsubUnboundedS…
R: @chamikaramj
comment created time in an hour
PR opened apache/beam
…ource transform for dataflow runner v2
[BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Choose reviewer(s) and mention them in a comment (
R: @username). - [ ] Format the pull request title like
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
| Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 |
|---|---|---|---|---|---|---|
| Go | --- | --- | --- | |||
| Java | ||||||
| Python | --- | --- | ||||
| XLang | --- | --- |
Pre-Commit Tests Status (on master branch)
| --- | Java | Python | Go | Website | Whitespace | Typescript |
|---|---|---|---|---|---|---|
| Non-portable | ||||||
| Portable | --- | --- | --- | --- | --- |
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
pr created time in an hour
pull request commentapache/beam
[BEAM-11695] Combiner packing in Dataflow
This was merged after release branch was cut. If this is a blocker for 2.28.0 please send in a cherry-pick to the release branch: https://github.com/apache/beam/tree/release-2.28.0
comment created time in an hour
Pull request review commentapache/beam
[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner
static void verifyDoFnSupportedStreaming(DoFn<?, ?> fn) { } static void verifyDoFnSupported(DoFn<?, ?> fn, boolean streaming) {- if (DoFnSignatures.usesSetState(fn)) {
Thanks for the support status confirmation! I would image we still want to keep this checking part for runner v2 as well.
comment created time in an hour
pull request commentapache/beam
[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner
Ah - good point. We still need to exclude those tests for the V2 runner.
On Wed, Jan 27, 2021 at 4:03 PM Reuven Lax relax@google.com wrote:
No - this PR adds support to the legacy runner only.
On Wed, Jan 27, 2021 at 3:52 PM Boyuan Zhang notifications@github.com wrote:
@boyuanzz commented on this pull request.
In runners/google-cloud-dataflow-java/build.gradle https://github.com/apache/beam/pull/13802#discussion_r565721856:
@@ -165,8 +163,6 @@ def commonLegacyExcludeCategories = [ def commonRunnerV2ExcludeCategories = [ 'org.apache.beam.sdk.testing.UsesCommittedMetrics', 'org.apache.beam.sdk.testing.UsesGaugeMetrics',
'org.apache.beam.sdk.testing.UsesSetState',Are MapState and SetState also supported in dataflow runner v2(unified worker) within this changes?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/apache/beam/pull/13802#pullrequestreview-577873503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAYJVNZAEBYVMPTRQAKZGDS4CRKZANCNFSM4WP3BTDA .
comment created time in an hour
pull request commentapache/beam
[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner
No - this PR adds support to the legacy runner only.
On Wed, Jan 27, 2021 at 3:52 PM Boyuan Zhang notifications@github.com wrote:
@boyuanzz commented on this pull request.
In runners/google-cloud-dataflow-java/build.gradle https://github.com/apache/beam/pull/13802#discussion_r565721856:
@@ -165,8 +163,6 @@ def commonLegacyExcludeCategories = [ def commonRunnerV2ExcludeCategories = [ 'org.apache.beam.sdk.testing.UsesCommittedMetrics', 'org.apache.beam.sdk.testing.UsesGaugeMetrics',
'org.apache.beam.sdk.testing.UsesSetState',Are MapState and SetState also supported in dataflow runner v2(unified worker) within this changes?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/apache/beam/pull/13802#pullrequestreview-577873503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAYJVNZAEBYVMPTRQAKZGDS4CRKZANCNFSM4WP3BTDA .
comment created time in an hour
pull request commentapache/beam
[BEAM-11689] Add public.nexus.pentaho.org to offline repositories.
Can you send a cherry-pick to 2.28.0 branch as well ?
comment created time in an hour
pull request commentapache/beam
[BEAM-11643] Add BigDecimal support for SpannerIO
2.28.0 release branch was already cut: https://github.com/apache/beam/tree/release-2.28.0 So assuming this is a blocker, can you propose a cherry-pick to that branch as well ?
comment created time in an hour
push eventapache/beam
commit sha a9d861273bc1c64a984c73209504461bd63fb4be
[BEAM-11679] Override PubsubUnboundedSource transform for dataflow runner v2
commit sha 56cebdacc8287ddd804d4e5abdbbcd54e6d4d597
Populate PubsubMessage message id
commit sha 02ec09c04f33cd2584db88a004ff26ac1840f771
Enable tests
commit sha 197d33802fb4eab3d1829ba122bfe1b43e16949f
Exclude failing FhirIO ITs
commit sha bc7930e60f6a8206273a3a8373e25d03085d3e61
Include runner v2 IT tests in java post commit
commit sha ebc3bc31e1f4185241e4a2cf2212772fd7c1fb66
Merge pull request #13797 from [BEAM-11679] Override PubsubUnboundedSource transform for dataflow runner v2 [BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…
push time in an hour
PR merged apache/beam
…nner v2
And also include RunnerV2 integration tests in post commit.
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Choose reviewer(s) and mention them in a comment (
R: @username). - [ ] Format the pull request title like
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
| Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 |
|---|---|---|---|---|---|---|
| Go | --- | --- | --- | |||
| Java | ||||||
| Python | --- | --- | ||||
| XLang | --- | --- |
Pre-Commit Tests Status (on master branch)
| --- | Java | Python | Go | Website | Whitespace | Typescript |
|---|---|---|---|---|---|---|
| Non-portable | ||||||
| Portable | --- | --- | --- | --- | --- |
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
pr closed time in an hour
pull request commentapache/beam
[BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…
Can you also propose a cherry-pick to 2.28.0 release branch ? (assuming this is a blocker)
comment created time in an hour
push eventapache/beam
commit sha 8070497559bafb5717c9f3d43e4d662a72d7d919
Add BigDecimal support for SpannerIO Since Numeric types were added to Spanner, it is possible to use BigDecimal values in Spanner mutations.
commit sha 9a4505e8de1b7e30e19a31eee1a18ca6e744351b
Merge pull request #13762: [BEAM-11643] Add BigDecimal support for SpannerIO
push time in an hour
PR merged apache/beam
Since Numeric types were added to Spanner, it is possible to use BigDecimal values in Spanner mutations.
This PR adds this support in the MutationSizeEstimator.
Support for writing these types was already added in #12818
| Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 |
|---|---|---|---|---|---|---|
| Go | --- | --- | --- | |||
| Java | ||||||
| Python | --- | --- | ||||
| XLang | --- | --- |
Pre-Commit Tests Status (on master branch)
| --- | Java | Python | Go | Website | Whitespace | Typescript |
|---|---|---|---|---|---|---|
| Non-portable | ||||||
| Portable | --- | --- | --- | --- | --- |
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
pr closed time in an hour
Pull request review commentapache/beam
[BEAM-11679] Override PubsubUnboundedSource transform for dataflow ru…
task javaPreCommitPortabilityApi() { task javaPostCommit() { dependsOn ":runners:google-cloud-dataflow-java:postCommit"+ dependsOn ":runners:google-cloud-dataflow-java:postCommitRunnerV2"
I like the idea that we add runner v2 post commit into current test suite. One further question I have is that should we consider have dataflow post commit separated from java post commit? cc: @tysonjh
comment created time in an hour
Pull request review commentapache/beam
[BEAM-1474]. Adding MapState and SetState support for the Dataflow runner
def commonLegacyExcludeCategories = [ def commonRunnerV2ExcludeCategories = [ 'org.apache.beam.sdk.testing.UsesCommittedMetrics', 'org.apache.beam.sdk.testing.UsesGaugeMetrics',- 'org.apache.beam.sdk.testing.UsesSetState',
Are MapState and SetState also supported in dataflow runner v2(unified worker) within this changes?
comment created time in an hour
PR closed apache/beam
This basically results multi-language pipelines using GCR images for external SDKs.
This update only works for released Beam versions. For unreleased Beam code SDK harness containers for external SDKs have to be manually overridden similar to today.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Choose reviewer(s) and mention them in a comment (
R: @username). - [ ] Format the pull request title like
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
| Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2 |
|---|---|---|---|---|---|---|
| Go | --- | --- | --- | |||
| Java | ||||||
| Python | --- | --- | ||||
| XLang | --- | --- |
Pre-Commit Tests Status (on master branch)
| --- | Java | Python | Go | Website | Whitespace | Typescript |
|---|---|---|---|---|---|---|
| Non-portable | ||||||
| Portable | --- | --- | --- | --- | --- |
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
pr closed time in 2 hours
pull request commentapache/beam
[BEAM-11613] Updates Dataflow pipelines to always use SDK Harness containers hosted in GCR
This PR is on hold till we finalize some other things in the Dataflow side. Will close for now and reopen when ready.
comment created time in 2 hours