profile
viewpoint
Matthias J. Sax mjsax @confluentinc Reno, NV Software Engineer | ksqlDB and Kafka Streams | Apache Committer and PMC member (Kafka, Flink, Storm)

mjsax/aeolus 15

Batching and optimization framework on top of Apache Storm.

mjsax/flink-external 15

Stuff built on top of Apache Flink

mbalassi/flink 9

Mirror of Apache Flink

mjsax/kafka-example 2

Simple example for reading and writing into Kafka

mjsax/datastructures 1

C++ library providing high efficient template-based datastructrues.

mjsax/flink-web 1

Mirror of Apache Flink Website

mjsax/performance 1

Flink performance tests

izzymsft/Getting-Started-with-Contributing-to-Apache-Kafka 0

Getting Started with Contributing to Apache Kafka

krichter722/lrb-validator 0

improvements of the Linear Road Benchmark validation tool at http://www.cs.brandeis.edu/~linearroad/tools.html

mjsax/DataflowJavaSDK 0

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

issue commentconfluentinc/kafka-streams-examples

Naming when adding global store

After a quick look into the code, it seems there is bug. I created https://issues.apache.org/jira/browse/KAFKA-13507

I guess, one workaround would be to use Topology#addGlobalStore instead:

StreamsBuilder builder = new StreamsBuilder();
Topology t = builder.build();
t.addGlobalStore(...)
blacksab

comment created time in 3 days

issue commentconfluentinc/kafka-streams-examples

Question: Adding example for reordering out of order messages

Should we still merge #441 or move directly to a kafka-tutorials contribution? No harm to merge, but the examples repo is superseded with the tutorial so not sure how much sense it makes to add to ti?

1123

comment created time in 3 days

push eventapache/kafka

Walker Carlson

commit sha 965ec40c0a282bc311c702c6eb795689ec7ddfa5

KAFKA-12648: Make changing the named topologies have a blocking option (#11479) Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Guozhang Wang <guozhang@confluent.io>

view details

push time in 3 days

PR merged apache/kafka

KAFKA-12648: Make changing the named topologies have a blocking option

Use a Kafka future to be able to add, remove then add back the same topology

Committer Checklist (excluded from commit message)

  • [ ] Verify design and implementation
  • [ ] Verify test coverage and CI build status
  • [ ] Verify documentation (including upgrade notes)
+452 -38

3 comments

7 changed files

wcarlson5

pr closed time in 3 days

Pull request review commentapache/kafka

KAFKA-13423: GlobalThread should not log ERROR on clean shutdown

 public void shouldGetToRunningWithOnlyGlobalTopology() throws Exception {         startStreams();         waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, Duration.ofSeconds(30)); -        kafkaStreams.close();

But it also seems like we are missing tests for the error case where the state of the Streams client closes to ERROR.

Not sure what this test should verify exactly? The final state of the client? Some log messages?

However, then we need a separate PR for the missing test or a ticket that documents that a test is missing.

Happy to do a follow up PR.

Have you had a look into GlobalStreamThreadTest? Could we add something there?

I don't think that GlobalStreamThreadTest works, because it tests the thread in isolation, while we want to test the interaction with KafkaStream client? (At least from my current understanding from you comment above). Maybe we could add a test to KafkaStreamsTest though?

Let me update this PR removing this test change so we can merge it, and work on a follow up PR afterwards.

mjsax

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentconfluentinc/ksql

chore: add deprecation warning/notice to left/outer joins without grace period

+/*+ * Copyright 2021 Confluent Inc.+ *+ * Licensed under the Confluent Community License (the "License"); you may not use+ * this file except in compliance with the License.  You may obtain a copy of the+ * License at+ *+ * http://www.confluent.io/confluent-community-license+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+ * WARRANTIES OF ANY KIND, either express or implied.  See the License for the+ * specific language governing permissions and limitations under the License.+ */++package io.confluent.ksql.rest.server.computation;++import static io.confluent.ksql.rest.server.computation.DeprecatedStatementsChecker.Deprecations.DEPRECATED_STREAM_STREAM_OUTER_JOIN_WITH_NO_GRACE;+import static org.hamcrest.MatcherAssert.assertThat;+import static org.hamcrest.Matchers.is;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++import com.google.common.collect.Iterables;+import io.confluent.ksql.execution.ddl.commands.KsqlTopic;+import io.confluent.ksql.execution.plan.JoinType;+import io.confluent.ksql.metastore.MetaStore;+import io.confluent.ksql.metastore.TypeRegistry;+import io.confluent.ksql.metastore.model.DataSource;+import io.confluent.ksql.metastore.model.KsqlStream;+import io.confluent.ksql.metastore.model.KsqlTable;+import io.confluent.ksql.name.SourceName;+import io.confluent.ksql.parser.AstBuilder;+import io.confluent.ksql.parser.DefaultKsqlParser;+import io.confluent.ksql.parser.KsqlParser;+import io.confluent.ksql.parser.tree.Statement;+import io.confluent.ksql.schema.ksql.LogicalSchema;+import org.junit.Before;+import org.junit.Test;+import org.junit.runner.RunWith;+import org.mockito.Mock;+import org.mockito.Mockito;+import org.mockito.junit.MockitoJUnitRunner;+import java.util.Optional;++@RunWith(MockitoJUnitRunner.class)+public class DeprecatedStatementsCheckerTest {+  private final KsqlParser STATEMENT_PARSER = new DefaultKsqlParser();++  private final SourceName STREAM_1 = SourceName.of("STREAM_1");+  private final SourceName STREAM_2 = SourceName.of("STREAM_2");+  private final SourceName STREAM_3 = SourceName.of("STREAM_3");+  private final SourceName TABLE_1 = SourceName.of("TABLE_1");++  private DeprecatedStatementsChecker statementsChecker;++  @Mock+  private MetaStore metaStore;++  @Before+  public void setup() {+    statementsChecker = new DeprecatedStatementsChecker(metaStore);++    final DataSource streamSource1 = newKsqlStream(STREAM_1);+    when(metaStore.getSource(STREAM_1)).thenReturn(streamSource1);++    final DataSource streamSource2 = newKsqlStream(STREAM_2);+    when(metaStore.getSource(STREAM_2)).thenReturn(streamSource2);++    final DataSource streamSource3 = newKsqlStream(STREAM_3);+    when(metaStore.getSource(STREAM_3)).thenReturn(streamSource3);++    final DataSource tableSource1 = newKsqlTable(TABLE_1);+    when(metaStore.getSource(TABLE_1)).thenReturn(tableSource1);+  }++  @Test+  public void shouldDeprecateStreamStreamJoinsWithNoGrace() {+    for (final JoinType joinType : JoinType.values()) {+      checkDeprecatedStatement(String.format(+          "SELECT * FROM %s AS l "+              + "%s JOIN %s AS r WITHIN 1 SECOND ON l.K = r.K;",+          STREAM_1.text(),+          (joinType == JoinType.OUTER) ? "FULL OUTER" : joinType,

Do we not support OUTER JOIN as keyword (similar to LEFT JOIN), but only FULL OUTER JOIN ?

In the end I think we should support all of those:

  • INNER JOIN or short JOIN
  • LEFT OUTER JOIN or short LEFT JOIN
  • RIGHT OUTER JOIN or short RIGHT JOIN (future work to add right-joins)
  • FULL OUTER JOIN or short OUTER JOIN

Would be follow up work thought to extend the grammar I guess...

spena

comment created time in 13 days

Pull request review commentconfluentinc/ksql

chore: add deprecation warning/notice to left/outer joins without grace period

+/*+ * Copyright 2021 Confluent Inc.+ *+ * Licensed under the Confluent Community License (the "License"); you may not use+ * this file except in compliance with the License.  You may obtain a copy of the+ * License at+ *+ * http://www.confluent.io/confluent-community-license+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+ * WARRANTIES OF ANY KIND, either express or implied.  See the License for the+ * specific language governing permissions and limitations under the License.+ */++package io.confluent.ksql.rest.server.computation;++import static io.confluent.ksql.rest.server.computation.DeprecatedStatementsChecker.Deprecations.DEPRECATED_STREAM_STREAM_OUTER_JOIN_WITH_NO_GRACE;+import static org.hamcrest.MatcherAssert.assertThat;+import static org.hamcrest.Matchers.is;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++import com.google.common.collect.Iterables;+import io.confluent.ksql.execution.ddl.commands.KsqlTopic;+import io.confluent.ksql.execution.plan.JoinType;+import io.confluent.ksql.metastore.MetaStore;+import io.confluent.ksql.metastore.TypeRegistry;+import io.confluent.ksql.metastore.model.DataSource;+import io.confluent.ksql.metastore.model.KsqlStream;+import io.confluent.ksql.metastore.model.KsqlTable;+import io.confluent.ksql.name.SourceName;+import io.confluent.ksql.parser.AstBuilder;+import io.confluent.ksql.parser.DefaultKsqlParser;+import io.confluent.ksql.parser.KsqlParser;+import io.confluent.ksql.parser.tree.Statement;+import io.confluent.ksql.schema.ksql.LogicalSchema;+import org.junit.Before;+import org.junit.Test;+import org.junit.runner.RunWith;+import org.mockito.Mock;+import org.mockito.Mockito;+import org.mockito.junit.MockitoJUnitRunner;+import java.util.Optional;++@RunWith(MockitoJUnitRunner.class)+public class DeprecatedStatementsCheckerTest {+  private final KsqlParser STATEMENT_PARSER = new DefaultKsqlParser();++  private final SourceName STREAM_1 = SourceName.of("STREAM_1");+  private final SourceName STREAM_2 = SourceName.of("STREAM_2");+  private final SourceName STREAM_3 = SourceName.of("STREAM_3");+  private final SourceName TABLE_1 = SourceName.of("TABLE_1");++  private DeprecatedStatementsChecker statementsChecker;++  @Mock+  private MetaStore metaStore;++  @Before+  public void setup() {+    statementsChecker = new DeprecatedStatementsChecker(metaStore);++    final DataSource streamSource1 = newKsqlStream(STREAM_1);+    when(metaStore.getSource(STREAM_1)).thenReturn(streamSource1);++    final DataSource streamSource2 = newKsqlStream(STREAM_2);+    when(metaStore.getSource(STREAM_2)).thenReturn(streamSource2);++    final DataSource streamSource3 = newKsqlStream(STREAM_3);+    when(metaStore.getSource(STREAM_3)).thenReturn(streamSource3);++    final DataSource tableSource1 = newKsqlTable(TABLE_1);+    when(metaStore.getSource(TABLE_1)).thenReturn(tableSource1);+  }++  @Test+  public void shouldDeprecateStreamStreamJoinsWithNoGrace() {+    for (final JoinType joinType : JoinType.values()) {+      checkDeprecatedStatement(String.format(+          "SELECT * FROM %s AS l "+              + "%s JOIN %s AS r WITHIN 1 SECOND ON l.K = r.K;",+          STREAM_1.text(),+          (joinType == JoinType.OUTER) ? "FULL OUTER" : joinType,+          STREAM_2.text()+      ));+    }+  }++  @Test+  public void shouldDeprecateStreamStreamJoinsWithNoGraceOnMultiStreamStreamJoinOrder1() {+    for (final JoinType joinType : JoinType.values()) {+      checkDeprecatedStatement(String.format(+          "SELECT * FROM %s AS l "+              + "%s JOIN %s AS m WITHIN 1 SECOND GRACE PERIOD 1 SECOND ON l.K = m.K "+              + "%s JOIN %s AS r WITHIN 1 SECOND ON m.K = r.K;",+          STREAM_1.text(),+          (joinType == JoinType.OUTER) ? "FULL OUTER" : joinType,+          STREAM_2.text(),+          (joinType == JoinType.OUTER) ? "FULL OUTER" : joinType,+          STREAM_3.text()+      ));+    }+  }++  @Test+  public void shouldDeprecateStreamStreamJoinsWithNoGraceOnMultiStreamStreamJoinOrder2() {+    for (final JoinType joinType : JoinType.values()) {+      checkDeprecatedStatement(String.format(+          "SELECT * FROM %s AS l "+              + "%s JOIN %s AS r WITHIN 1 SECOND ON m.K = r.K "+              + "%s JOIN %s AS m WITHIN 1 SECOND GRACE PERIOD 1 SECOND ON l.K = m.K;",+          STREAM_1.text(),+          (joinType == JoinType.OUTER) ? "FULL OUTER" : joinType,+          STREAM_2.text(),+          (joinType == JoinType.OUTER) ? "FULL OUTER" : joinType,+          STREAM_3.text()+      ));+    }+  }

Can we add a third case with two joins both using GRACE and thus no warning should surface?

spena

comment created time in 13 days

Pull request review commentconfluentinc/ksql

chore: add deprecation warning/notice to left/outer joins without grace period

+/*+ * Copyright 2021 Confluent Inc.+ *+ * Licensed under the Confluent Community License (the "License"); you may not use+ * this file except in compliance with the License.  You may obtain a copy of the+ * License at+ *+ * http://www.confluent.io/confluent-community-license+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+ * WARRANTIES OF ANY KIND, either express or implied.  See the License for the+ * specific language governing permissions and limitations under the License.+ */++package io.confluent.ksql.rest.server.computation;++import static io.confluent.ksql.analyzer.Analysis.AliasedDataSource;++import io.confluent.ksql.engine.rewrite.DataSourceExtractor;+import io.confluent.ksql.metastore.MetaStore;+import io.confluent.ksql.metastore.model.DataSource;+import io.confluent.ksql.parser.tree.CreateAsSelect;+import io.confluent.ksql.parser.tree.Join;+import io.confluent.ksql.parser.tree.JoinedSource;+import io.confluent.ksql.parser.tree.Query;+import io.confluent.ksql.parser.tree.Relation;+import io.confluent.ksql.parser.tree.Statement;+import io.confluent.ksql.parser.tree.WithinExpression;+import java.util.Objects;+import java.util.Optional;++public class DeprecatedStatementsChecker {+  public enum Deprecations {+    DEPRECATED_STREAM_STREAM_OUTER_JOIN_WITH_NO_GRACE(

Nit: drop OUTER in name?

spena

comment created time in 13 days

Pull request review commentconfluentinc/ksql

chore: add deprecation warning/notice to left/outer joins without grace period

+/*+ * Copyright 2021 Confluent Inc.+ *+ * Licensed under the Confluent Community License (the "License"); you may not use+ * this file except in compliance with the License.  You may obtain a copy of the+ * License at+ *+ * http://www.confluent.io/confluent-community-license+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+ * WARRANTIES OF ANY KIND, either express or implied.  See the License for the+ * specific language governing permissions and limitations under the License.+ */++package io.confluent.ksql.rest.server.computation;++import static io.confluent.ksql.analyzer.Analysis.AliasedDataSource;++import io.confluent.ksql.engine.rewrite.DataSourceExtractor;+import io.confluent.ksql.metastore.MetaStore;+import io.confluent.ksql.metastore.model.DataSource;+import io.confluent.ksql.parser.tree.CreateAsSelect;+import io.confluent.ksql.parser.tree.Join;+import io.confluent.ksql.parser.tree.JoinedSource;+import io.confluent.ksql.parser.tree.Query;+import io.confluent.ksql.parser.tree.Relation;+import io.confluent.ksql.parser.tree.Statement;+import io.confluent.ksql.parser.tree.WithinExpression;+import java.util.Objects;+import java.util.Optional;++public class DeprecatedStatementsChecker {+  public enum Deprecations {+    DEPRECATED_STREAM_STREAM_OUTER_JOIN_WITH_NO_GRACE(+        "DEPRECATION NOTICE: Left/Outer stream-stream joins statements without a GRACE PERIOD "+            + "will not be accepted in a future ksqlDB version.\n"+            + "Please use the GRACE PERIOD clause as specified in "+            + "https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/"+            + "select-push-query/"+    );++    private final String noticeMessage;++    Deprecations(final String noticeMessage) {+      this.noticeMessage = noticeMessage;+    }++    public String getNoticeMessage() {+      return noticeMessage;+    }+  }++  final MetaStore metaStore;++  DeprecatedStatementsChecker(final MetaStore metaStore) {+    this.metaStore = Objects.requireNonNull(metaStore, "metaStore");+  }++  public Optional<Deprecations> checkStatement(final Statement statement) {+    if (isStreamStreamJoinWithoutGraceStatement(statement)) {+      return Optional.of(Deprecations.DEPRECATED_STREAM_STREAM_OUTER_JOIN_WITH_NO_GRACE);+    }++    return Optional.empty();+  }++  private boolean isStreamStreamJoinWithoutGraceStatement(final Statement statement) {+    if (statement instanceof CreateAsSelect) {+      return isStreamStreamJoinWithoutGraceQuery(((CreateAsSelect) statement).getQuery());+    } else if (statement instanceof Query) {+      return isStreamStreamJoinWithoutGraceQuery((Query) statement);+    }++    return false;+  }++  private boolean isStreamStreamJoinWithoutGraceQuery(final Query query) {+    if (query.getFrom() instanceof Join) {+      final Join join = (Join) query.getFrom();++      // check left joined source is a stream+      if (!isJoinedSourceStream(join.getLeft())) {+        return false;+      }++      for (final JoinedSource joinedSource : join.getRights()) {+        // check right joined source is a stream+        if (!isJoinedSourceStream(joinedSource.getRelation())) {+          break;+        }++        if (!joinedSource.getWithinExpression().flatMap(WithinExpression::getGrace).isPresent()) {+          return true;+        }+      }+    }++    return false;+  }++  private boolean isJoinedSourceStream(final Relation relation) {+    // DataSourceExtractor must be initialized everytime we need to extract data from a node+    // because it changes its internal state to keep all sources found+    final DataSourceExtractor dataSourceExtractor = new DataSourceExtractor(metaStore, false);++    // The relation object should have only one joined source, but the extract sources method+    // returns a list. This loop checks all returned values are a Stream just to prevent throwing+    // an exception if more than one value is returned.+    for (final AliasedDataSource source : dataSourceExtractor.extractDataSources(relation)) {+      if (source.getDataSource().getDataSourceType() != DataSource.DataSourceType.KSTREAM) {+        return false;

Not sure if I can follow? If we have more than one join, the input relation could be a Join itself, right? So we would need to to recursively step down, until we hit a Join for which both inputs are not a Join (ie, until the input relation to this method is not a Join)?

spena

comment created time in 13 days

Pull request review commentconfluentinc/ksql

chore: add deprecation warning/notice to left/outer joins without grace period

+/*+ * Copyright 2021 Confluent Inc.+ *+ * Licensed under the Confluent Community License (the "License"); you may not use+ * this file except in compliance with the License.  You may obtain a copy of the+ * License at+ *+ * http://www.confluent.io/confluent-community-license+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+ * WARRANTIES OF ANY KIND, either express or implied.  See the License for the+ * specific language governing permissions and limitations under the License.+ */++package io.confluent.ksql.rest.server.computation;++import static io.confluent.ksql.analyzer.Analysis.AliasedDataSource;++import io.confluent.ksql.engine.rewrite.DataSourceExtractor;+import io.confluent.ksql.metastore.MetaStore;+import io.confluent.ksql.metastore.model.DataSource;+import io.confluent.ksql.parser.tree.CreateAsSelect;+import io.confluent.ksql.parser.tree.Join;+import io.confluent.ksql.parser.tree.JoinedSource;+import io.confluent.ksql.parser.tree.Query;+import io.confluent.ksql.parser.tree.Relation;+import io.confluent.ksql.parser.tree.Statement;+import io.confluent.ksql.parser.tree.WithinExpression;+import java.util.Objects;+import java.util.Optional;++public class DeprecatedStatementsChecker {+  public enum Deprecations {+    DEPRECATED_STREAM_STREAM_OUTER_JOIN_WITH_NO_GRACE(+        "DEPRECATION NOTICE: Left/Outer stream-stream joins statements without a GRACE PERIOD "

Drop Left/Outer -- we make GRACE PERIOD mandatory for INNER, too, right?

spena

comment created time in 13 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentconfluentinc/ksql

chore: add schema id to format property

 public void shouldThrowWhenCreatingFromUnsupportedProperty() {   @Test   public void shouldNotThrowWhenCreatingFromSupportedProperty() {     // Given:-    final FormatInfo formatInfo = FormatInfo.of("AVRO", ImmutableMap.of("fullSchemaName", "1"));+    final FormatInfo formatInfo = FormatInfo.of("AVRO", ImmutableMap.of("schemaId", "1", "fullSchemaName", "2"));+    final FormatInfo protobufInfo = FormatInfo.of("PROTOBUF",+        ImmutableMap.of("schemaId", "1"));+    final FormatInfo jsonSRFormatInfo = FormatInfo.of("JSON_SR",+        ImmutableMap.of("schemaId", "123"));

Is "fullSchemaName" missing for protobuf and json?

lihaosky

comment created time in 5 days

Pull request review commentconfluentinc/ksql

chore: add schema id to format property

 public void shouldThrowWhenCreatingFromUnsupportedProperty() {   @Test   public void shouldNotThrowWhenCreatingFromSupportedProperty() {     // Given:-    final FormatInfo formatInfo = FormatInfo.of("AVRO", ImmutableMap.of("fullSchemaName", "1"));+    final FormatInfo formatInfo = FormatInfo.of("AVRO", ImmutableMap.of("schemaId", "1", "fullSchemaName", "2"));+    final FormatInfo protobufInfo = FormatInfo.of("PROTOBUF",

nit: protobufFormatInfo ?

lihaosky

comment created time in 5 days

Pull request review commentconfluentinc/ksql

chore: add schema id to format property

 public void shouldThrowWhenCreatingFromUnsupportedProperty() {   @Test   public void shouldNotThrowWhenCreatingFromSupportedProperty() {     // Given:-    final FormatInfo formatInfo = FormatInfo.of("AVRO", ImmutableMap.of("fullSchemaName", "1"));+    final FormatInfo formatInfo = FormatInfo.of("AVRO", ImmutableMap.of("schemaId", "1", "fullSchemaName", "2"));

nit: avroFormatInfo ?

lihaosky

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentconfluentinc/ksql

stream to table join emit changes when table updates

Since v0.19.0 FK-joins are supported: https://www.confluent.io/blog/ksqldb-0-19-adds-data-modeling-foreign-key-joins/ (You actually linked to the ticket above).

For this case, you don't need to change your left input table containing the composite key: FK-joins don't require that input tables are co-partition, but the runtime will take care of it internally. If you are interested how FK-joins work at the runtime level, check out https://www.confluent.io/blog/data-enrichment-with-kafka-streams-foreign-key-joins/

However, for your case you will need to use v0.21.0 that adds support for expressions in the join condition: https://www.confluent.io/blog/announcing-ksqldb-0-21-new-features-major-upgrades/

Thus, you simple join query

select d.rowkey, d.id, d.code from detail_table d inner join header_table h on struct(id:=d.rowkey->id) = h.rowkey emit changes;

should actually just work using v0.21.0.

msillence

comment created time in 5 days

issue commentconfluentinc/kafka-streams-examples

Question: Adding example for reordering out of order messages

PR: https://github.com/confluentinc/kafka-streams-examples/pull/411

1123

comment created time in 5 days

issue commentconfluentinc/ksql

problem in creating table/stream with key or primary key

CREATE stream SubscriptionCreatedInbound(struct<...> KEY)...

Seems the column name is missing?

CREATE stream SubscriptionCreatedInbound(myKey struct<...> KEY)...
unknownBangalore

comment created time in 5 days

issue commentconfluentinc/ksql

The key format is invalid when create stream with syntax 'create stream as select'

You are setting the format for both key and value to KAFKA via WITH(...FORMAT='KAFKA'), so it won't be AVRO. (Not sure what format you expect?)

What I am wondering though is, why you would get any data in the result stream key field:

...AS SELECT CAST(NULL AS VARCHAR) FROM...

produce an output stream with a single column, and this single column should be a "value column". I would assume that the key would be set to null?

VisionaryAries

comment created time in 5 days

issue commentconfluentinc/kafka-streams-examples

Naming when adding global store

You can use Consumed.as("name").withKeySerde(...).withValueSerde(...) or Consumed.with(...).withName("name").

blacksab

comment created time in 5 days

push eventmjsax/kafka

Chris Egerton

commit sha f875576f218c90bc4a682a641ff149d66f75e0b8

KAFKA-13469: Block for in-flight record delivery before end-of-life source task offset commit (#11524) Although committing source task offsets without blocking on the delivery of all in-flight records is beneficial most of the time, it can lead to duplicate record delivery if there are in-flight records at the time of the task's end-of-life offset commit. A best-effort attempt is made here to wait for any such in-flight records to be delivered before proceeding with the end-of-life offset commit for source tasks. Connect will block for up to offset.flush.timeout.ms milliseconds before calculating the latest committable offsets for the task and flushing those to the persistent offset store. Author: Chris Egerton <chrise@confluent.io> Reviewer: Randall Hauch <rhauch@gmail.com>

view details

Bruno Cadonna

commit sha d1e0d2b474b0bdb0b0141b6d341ce77dd331a8d4

HOTFIX: Set version of jgit to avoid unsupported version error (#11554) A new version of JGit that is used by grgit that is used by gradle causes the following error: org/eclipse/jgit/storage/file/FileRepositoryBuilder has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 The reason is that version 6.0.0.202111291000-r of JGrit was compiled with a newer Java version than Java 8, probably Java 11. Explicitly setting the version of JGrit in gradle to 5.12.0.202106070339-r fixes the issue. Reviewers: David Jacot <djacot@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Alexander Stohr, David Arthur <mumrah@gmail.com>

view details

loboya~

commit sha 42306ba267aba684f7377bd78db90fdc4139546f

KAFKA-12932: Interfaces for SnapshotReader and SnapshotWriter (#11529) Change the snapshot API so that SnapshotWriter and SnapshotReader are interfaces. Change the existing types SnapshotWriter and SnapshotReader to use a different name and to implement the interfaces introduced by this commit. Co-authored-by: loboxu <loboxu@tencent.com> Reviews: José Armando García Sancio <jsancio@users.noreply.github.com>

view details

push time in 6 days

PullRequestReviewEvent

push eventmjsax/ksql

Zara Lim

commit sha 25c9ca5f1f59136f574b261be43eb09947af264b

fix: port changes from #7863 to 5.4.x (#8409) * fix: port changes from #7863 to 5.4.x * remove two lines

view details

Zara Lim

commit sha a1e5f8cfeb375af559d761ffdfb7f49d3cf9897c

Merge branch '5.4.x' into 5.5.x

view details

Zara Lim

commit sha 719f8d7eb6b6f35eeb566f65c39de242f8f653d8

Merge branch '5.5.x' into 6.0.x

view details

Zara Lim

commit sha a5bce2d872bf3bda8fc08c60440673b387a10e16

Merge branch '6.0.x' into 6.1.x

view details

Zara Lim

commit sha 8b4aa55857d64461879d199e032656b1e0bdec61

Merge branch '6.1.x' into 6.2.x

view details

Zara Lim

commit sha d89a9af9fd5bf44b4288df69a0e10c8ff469f55d

Merge branch '6.2.x' into 7.0.x

view details

Zara Lim

commit sha eafb3773b145c788214b9baff4cb14f81c71b70f

Merge branch '7.0.x' into 7.1.x

view details

Zara Lim

commit sha 5d94e50d55f88d2fe6733ccdd09fadc369dbd9e5

Merge branch '7.1.x'

view details

nateab

commit sha 5dd40134f6ad892cf092fd29ac2b0a3f60f37868

chore: bump command version from 11 to 12 for v0.24 (#8408)

view details

Patrick Stuedi

commit sha ed50b05ff7789df2afc7e651b8a3c61ffeefed5a

fix: synchronizing writes to localcommand file (#8406)

view details

Vicky Papavasileiou

commit sha aa7c1468693fc36f266de0b25bfca8d6519a5032

chore: flaky test (#8405) * chore: flaky test * assert eventually

view details

Hao Li

commit sha d6184b0569195b148e442f28db39c8de8e91335e

chore: add raw schema to session override (#8399) * chore: add raw schema to session override * fix: change name and add more test

view details

Patrick Stuedi

commit sha c4cc6d5c58dcbc82d2259192cea70eb2ae65ffba

fix: tolerate out-of-order execution on HARouting tests (#8415)

view details

Jim Galasyn

commit sha 22a0da45f6ee7a1b00b783a3a4b66edea6a2a013

DOCS-9865: Bump version token in 0.23.0 docs (#8424)

view details

Alan Sheinberg

commit sha fbbc076a9e2471744ae47e96511ea4cf09c52540

fix: Fixes flaky ScalablePushBandwidthThrottleIntegrationTest caused by timeout (#8410) * fix: Fixes flaky ScalablePushBandwidthThrottleIntegrationTest caused by timeout

view details

Alan Sheinberg

commit sha c24e3b344b9f5e316d8ef48d41b1aa8e1ba020fa

refactor: Consolidates the logic for launching queries in QueryExecutor (#8392) * refactor: Consolidates the logic for launching queries in QueryExecutor

view details

Zara Lim

commit sha fee904e07802195cd64432454d5e05dd6fb87099

fix: make to_bytes work with lowercase hex (#8423) * fix: make to_bytes work with lowercase hex * add another test

view details

Zara Lim

commit sha e030f106b76338d13675d68169936ce891e87594

feat: populate header columns (#8416) * feat: populate header columns * refactor a little * checkstyle * checkstyle * refactor

view details

push time in 6 days

push eventmjsax/kafka

Kirk True

commit sha 7b379539a5533179b0c86adb352548ac1aa82006

KAFKA-13202: KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC (#11284) This task is to provide a concrete implementation of the interfaces defined in KIP-255 to allow Kafka to connect to an OAuth/OIDC identity provider for authentication and token retrieval. While KIP-255 provides an unsecured JWT example for development, this will fill in the gap and provide a production-grade implementation. The OAuth/OIDC work will allow out-of-the-box configuration by any Apache Kafka users to connect to an external identity provider service (e.g. Okta, Auth0, Azure, etc.). The code will implement the standard OAuth client credentials grant type. The proposed change is largely composed of a pair of AuthenticateCallbackHandler implementations: one to login on the client and one to validate on the broker. See the following for more detail: KIP-768 KAFKA-13202 Reviewers: Yi Ding <dingyi.zj@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

view details

Colin Patrick McCabe

commit sha af8100b94fda4a27511797233e9845078ae8a69f

KAFKA-13340: Change ZooKeeperTestHarness to QuorumTestHarness (#11417) Change ZooKeeperTestHarness to QuorumTestHarness so that integration tests which inherit from this class can test Kafka in both ZK and KRaft mode. Test cases which do this can specify the modes they support by including a ParameterizedTest annotation before each test case, like the following: @ParameterizedTest @valuesource(strings = Array("zk", "kraft")) def testValidCreateTopicsRequests(quorum: String): Unit = { ... } For each value that is specified here (zk, kraft), the test case will be run once in the appropriate mode. So the test shown above is run twice. This allows integration tests to be incrementally converted over to support KRaft mode, rather than rewritten to support it. For now, test cases which do not specify a quorum argument will continue to run only in ZK mode. JUnit5 makes the quorum annotation visible in the TestInfo object which each @BeforEeach function in a test can optionally take. Therefore, this PR converts over the setUp function of the quorum base class, plus every derived class, to take a TestInfo argument. The TestInfo object gets "passed up the chain" to the base class, where it determines which quorum type we create (ZK or KRaft). In a few cases, I discovered test cases inheriting from the test harness that had more than one @BeforeEach function. Because the JUnit5 framework does not define the order in which @BeforeEach hooks are run, I changed these to overload setUp() instead, to avoid undefined behavior. The general approach taken here is to make as much as possible work with KRaft, but to leave some things as ZK-only when appropriate. For example, a test that explicitly requests an AdminZkClient object will get an exception if it is running in KRaft mode. Similarly, tests which explicitly request KafkaServer rather than KafkaBroker will get an exception when running in KRaft mode. As a proof of concept, this PR converts over kafka.api.MetricsTest to support KRaft. This PR also renames the quorum controller event handler thread to include the text "QuorumControllerEventHandler". This allows QuorumTestHarness to check for hanging quorum controller threads, as it does for hanging ZK-based controller threads. Finally, ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup caused many failing test runs. Therefore, I disabled it here and filed KAFKA-13421 to fix the test logic to be more reliable. Reviewers: Jason Gustafson <jason@confluent.io>, Igor Soarez <soarez@apple.com>

view details

David Jacot

commit sha 3aef0a5cebe3d17ed17a79b999d5f78a03b44949

MINOR: Bump trunk to 3.2.0-SNAPSHOT (#11458) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>

view details

Luizfrf3

commit sha 252a40ea1f273a5f11ccb5a9b362bceca7e66271

KAFKA-8941: Add RocksDB Metrics that Could not be Added due to RocksD… (#11441) This PR adds some RocksDB metrics that could not be added in KIP-471 due to RocksDB version. The new metrics are extracted using Histogram data provided by RocksDB API, and the old ones were extracted using Tickers. The new metrics added are memtable-flush-time-(avg|min|max) and compaction-time-(avg|min|max). Reviewer: Bruno Cadonnna <cadonna@apache.org>

view details

Lee Dongjin

commit sha 22d056c9b76c9bf8417d8701594d1fcee1c6a655

TRIVIAL: Fix type inconsistencies, unthrown exceptions, etc (#10678) Reviewers: Ismael Juma <ismael@juma.me.uk>, Bruno Cadonna <cadonna@apache.org>

view details

Victoria Xia

commit sha 01e6a6ebf2af87e12b60fb7cf55d261ded72ed3e

KAFKA-13261: Add support for custom partitioners in foreign key joins (#11368) Implements KIP-775. Co-authored-by: Tomas Forsman <tomas-forsman@users.noreply.github.com>

view details

Lee Dongjin

commit sha 9d305c79d312dbafafcb2a5fb936f6a0546c8aaa

KAFKA-13430: Remove broker-wide quota properties from the documentation (#11463) Reviewers: David Jacot <djacot@confluent.io>

view details

Luke Chen

commit sha 43bcc5682da82a602a4c0a000dc7433d0507b450

KAFKA-13396: Allow create topic without partition/replicaFactor (#11429) [KIP-464](https://cwiki.apache.org/confluence/display/KAFKA/KIP-464%3A+Defaults+for+AdminClient%23createTopic) (PR: https://github.com/apache/kafka/pull/6728) made it possible to create topics without passing partition count and/or replica factor when using the admin client. We incorrectly disallowed this via https://github.com/apache/kafka/pull/10457 while trying to ensure validation was consistent between ZK and the admin client (in this case the inconsistency was intentional). Fix this regression and add tests for the command lines in quick start (i.e. create topic and describe topic) to make sure it won't be broken in the future. Reviewers: Lee Dongjin <dongjin@apache.org>, Ismael Juma <ismael@juma.me.uk>

view details

Chris Egerton

commit sha c1bdfa125d224d07290a2d33cdac9d7387796fef

KAFKA-12226: Commit source task offsets without blocking on batch delivery (#11323) Replaces the current logic for committing source offsets, which is batch-based and blocks until the entirety of the current batch is fully written to and acknowledged by the broker, with a new non-blocking approach that commits source offsets for source records that have been "fully written" by the producer. The new logic consider a record fully written only if that source record and all records before it with the same source partition have all been written to Kafka and acknowledged. This new logic uses a deque for every source partition that a source task produces records for. Each element in that deque is a SubmittedRecord with a flag to track whether the producer has ack'd the delivery of that source record to Kafka. Periodically, the worker (on the same thread that polls the source task for records and transforms, converts, and dispatches them to the producer) polls acknowledged elements from the beginning of each of these deques and collects the latest offsets from these elements, storing them in a snapshot that is then committed on the separate source task offset thread. The behavior of the `offset.flush.timeout.ms property` is retained, but essentially now only applies to the actual writing of offset data to the internal offsets topic (if running in distributed mode) or the offsets file (if running in standalone mode). No time is spent during `WorkerSourceTask::commitOffsets` waiting on the acknowledgment of records by the producer. This behavior also does not change how the records are dispatched to the producer nor how the producer sends or batches those records. It's possible that memory exhaustion may occur if, for example, a single Kafka partition is offline for an extended period. In cases like this, the collection of deques in the SubmittedRecords class may continue to grow indefinitely until the partition comes back online and the SubmittedRecords in those deques that targeted the formerly-offline Kafka partition are acknowledged and can be removed. Although this may be suboptimal, it is no worse than the existing behavior of the framework in these cases. Author: Chris Egerton <chrise@confluent.io> Reviewed: Randall Hauch <rhauch@gmail.com>

view details

Jorge Esteban Quilcate Otoya

commit sha 807c5b4d282e7a7a16d0bb94aa2cda9566a7cc2d

KAFKA-10543: Convert KTable joins to new PAPI (#11412) * Migrate KTable joins to new Processor API. * Migrate missing KTableProcessorSupplier implementations. * Replace KTableProcessorSupplier with new Processor API implementation. Reviewers: John Roesler <vvcephei@apache.org>

view details

Niket

commit sha feee65a2cf1efed02422fe2c9392682b0aaf7882

MINOR: Adding a constant to denote UNKNOWN leader in LeaderAndEpoch (#11477) Reviewers: José Armando García Sancio <jsancio@gmail.com>, Jason Gustafson <jason@confluent.io>

view details

Jason Gustafson

commit sha 78a5e921d4566f2e2d56714834e3c5dd6ad17125

KAFKA-13417; Ensure dynamic reconfigurations set old config properly (#11448) This patch fixes a bug in `DynamicBrokerConfig` which causes some configuration changes to be ignored. In particular, the bug is the result of the reference to the old configuration getting indirectly mutated prior to the call to `BrokerReconfigurable.reconfigure`. This causes the first dynamic configuration update to pass effectively the same configuration as both `oldConfig` and `newConfig`. In cases such as in `DynamicThreadPool`, the update is ignored because the old configuration value matches the new configuration value. This bug only affects KRaft. It is protected in the zk broker by the call to `DynamicBrokerConfig.initialize()`, which overwrites the stored reference to the original configuration. The patch fixes the problem by ensuring that `initialize()` is also invoked in KRaft when `BrokerServer` starts up. Reviewers: David Jacot <djacot@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>

view details

Ismael Juma

commit sha b8c3a67012218188330110ce94636b70116131f9

MINOR: Remove topic null check from `TopicIdPartition` and adjust constructor order (#11403) `TopicPartition` allows a null `topic` and there are cases where we have a topic id, but no topic name. Even for `TopicIdPartition`, the non null topic name check was only enforced in one constructor. Also adjust the constructor order to move the nullable parameter to the end, update tests and javadoc. Reviewers: David Jacot <djacot@confluent.io>, Luke Chen <showuon@gmail.com>

view details

Jason Gustafson

commit sha 79d97bd29d059e8ba8ee7726b49d76e03e281059

MINOR: Log client disconnect events at INFO level (#11449) Client disconnects are crucial events for debugging. The fact that we only log them at DEBUG/TRACE means we rarely have them available outside of a testing context. This patch therefore increases verbosity to INFO level. In practice, we already have backoff configurations which should prevent these logs from getting too spammy. Reviewers: Guozhang Wang <wangguoz@gmail.com>

view details

Chris Egerton

commit sha 39b1bf4ee3cb63a889eb64d51d922866d37e7f5f

KAFKA-12487: Add support for cooperative consumer protocol with sink connectors (#10563) Currently, the `WorkerSinkTask`'s consumer rebalance listener (and related logic) is hardcoded to assume eager rebalancing, which means that all partitions are revoked any time a rebalance occurs and then the set of partitions included in `onPartitionsAssigned` is assumed to be the complete assignment for the task. Not only does this cause failures when the cooperative consumer protocol is used, it fails to take advantage of the benefits provided by that protocol. These changes alter framework logic to not only not break when the cooperative consumer protocol is used for a sink connector, but to reap the benefits of it as well, by not revoking partitions unnecessarily from tasks just to reopen them immediately after the rebalance has completed. This change will be necessary in order to support [KIP-726](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248), which currently proposes that the default consumer partition assignor be changed to the `CooperativeStickyAssignor`. Two integration tests are added to verify sink task behavior with both eager and cooperative consumer protocols, and new and existing unit tests are adopted as well. Reviewers: Nigel Liang <nigel@nigelliang.com>, Konstantine Karantasis <k.karantasis@gmail.com>

view details

A. Sophie Blee-Goldman

commit sha 908a6d2ad7807642b917231769c8d3b6a0c1ad16

KAFKA-12648: introduce TopologyConfig and TaskConfig for topology-level overrides (#11272) Most configs that are read and used by Streams today originate from the properties passed in to the KafkaStreams constructor, which means they get applied universally across all threads, tasks, subtopologies, and so on. The only current exception to this is the topology.optimization config which is parsed from the properties that get passed in to StreamsBuilder#build. However there are a handful of configs that could also be scoped to the topology level, allowing users to configure each NamedTopology independently of the others, where it makes sense to do so. This PR refactors the handling of these configs by interpreting the values passed in via KafkaStreams constructor as the global defaults, which can then be overridden for individual topologies via the properties passed in when building the NamedTopology. More topology-level configs may be added in the future, but this PR covers the following: max.task.idle.ms task.timeout.ms buffered.records.per.partition default.timestamp.extractor.class default.deserialization.exception.handler Reviewers: Guozhang Wang <wangguoz@gmail.com>, Walker Carlson <wcarlson@confluent.io>, Luke Chen <showuon@confluent.io>

view details

Jorge Esteban Quilcate Otoya

commit sha 214b59b3ec2c19bd1831076970aff86c1755d0e8

KAFKA-13429: ignore bin on new modules (#11415) Reviewers: John Roesler <vvcephei@apache.org>

view details

David Jacot

commit sha 6f6248509c0c38dcd867b28e6481de5370c6e082

MINOR: Introduce `ApiKeyVersionsSource` annotation for `ParameterizedTest` (#11468) It is common in our code base to have unit tests which must be run for all the versions of a given request. Most of the time, we do so by iterating over all the versions in the test itself which is error prone. With JUnit5 and ParameterizedTest, we can now use a custom arguments source for this case, which is way more convenient. It looks likes this: ``` @ParameterizedTest @ApiKeyVersionsSource(apiKey = ApiKeys.ADD_PARTITIONS_TO_TXN) public void mytest(short version) { // do smth based on version... } ``` This patch introduces the new annotation and updates `AddPartitionsToTxnRequestTest` test as a first example. I will migrate all the other cases in subsequent PRs. Reviewers: Luke Chen <showuon@gmail.com>, Jason Gustafson <jason@confluent.io>

view details

Jason Gustafson

commit sha e9db5a11e48ffeec88bb37feb5bc08b756bb6662

KAFKA-13421; Reenable `testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup` (#11485) This test was disabled in https://github.com/apache/kafka/commit/af8100b94fda4a27511797233e9845078ae8a69f. The reason the test was failing is that it assumes that the reference to `servers` can be mutated directly. The implementation in `IntegrationTestHarness` is intended to allow this by returning a mutable buffer, but the implementation actually returns a copy of the underlying collection. This caused the test case to create multiple `KafkaServer` instances instead of one as intended because it was modifying the copy. This led to the broker registration failure. Reviewers: David Jacot <djacot@confluent.io>

view details

Justine Olshan

commit sha e8818e234a879d5ca45accba0121f43f45381f4a

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs (#11331) With the changes for topic IDs, we have a different flow. When a broker receives a request, it uses a map to convert the topic ID to topic names. If the topic ID is not found in the map, we return a top level error and close the session. This decision was motivated by the difficulty to store “unresolved” partitions in the session. In earlier iterations we stored an “unresolved” partition object in the cache, but it was somewhat hard to reason about and required extra logic to try to resolve the topic ID on each incremental request and add to the session. It also required extra logic to forget the topic (either by topic ID if the topic name was never known or by topic name if it was finally resolved when we wanted to remove from the session.) One helpful simplifying factor is that we only allow one type of request (uses topic ID or does not use topic ID) in the session. That means we can rely on a session continuing to have the same information. We don’t have to worry about converting topics only known by name to topic ID for a response and we won’t need to convert topics only known by ID to name for a response. This PR introduces a change to store the "unresolved partitions" in the cached partition object. If a version 13+ request is sent with a topic ID that is unknown, a cached partition will be created with that fetch request data and a null topic name. On subsequent incremental requests, unresolved partitions may be resolved with the new IDs found in the metadata cache. When handling the request, getting all partitions will return a TopicIdPartition object that will be used to handle the request and build the response. Since we can rely on only one type of request (with IDs or without), the cached partitions map will have different keys depending on what fetch request version is being used. This PR involves changes both in FetchSessionHandler and FetchSession. Some major changes are outlined below. 1. FetchSessionHandler: Forgetting a topic and adding a new topic with the same name - We may have a case where there is a topic foo with ID 1 in the session. Upon a subsequent metadata update, we may have topic foo with ID 2. This means that topic foo has been deleted and recreated. When sending fetch requests version 13+ we will send a request to add foo ID 2 to the session and remove foo ID 1. Otherwise, we will fall back to the same behavior for versions 12 and below 2. FetchSession: Resolving in Incremental Sessions - Incremental sessions contain two distinct sets of partitions. Partitions that are sent in the latest request that are new/updates/forgotten partitions and the partitions already in the session. If we want to resolve unknown topic IDs we will need to handle both cases. * Partitions in the request - These partitions are either new or updating/forgetting previous partitions in the session. The new partitions are trivial. We either have a resolved partition or create a partition that is unresolved. For the other cases, we need to be a bit more careful. * For updated partitions we have a few cases – keep in mind, we may not programmatically know if a partition is an update: 1. partition in session is resolved, update is resolved: trivial
 2. partition in session is unresolved, update is unresolved: in code, this is equivalent to the case above, so trivial as well
 3. partition in session is unresolved, update is resolved: this means the partition in the session does not have a name, but the metadata cache now contains the name – to fix this we can check if there exists a cached partition with the given ID and update it both with the partition update and with the topic name.
 4. partition in session is resolved, update is unresolved: this means the partition in the session has a name, but the update was unable to be resolved (ie, the topic is deleted) – this is the odd case. We will look up the partition using the ID. We will find the old version with a name but will not replace the name. This will lead to an UNKNOWN_TOPIC_OR_PARTITION or INCONSISTENT_TOPIC_ID error which will be handled with a metadata update. Likely a future request will forget the partition, and we will be able to do so by ID.
 5. Two partitions in the session have IDs, but they are different: only one topic ID should exist in the metadata at a time, so likely only one topic ID is in the fetch set. The other one should be in the toForget. We will be able to remove this partition from the session. If for some reason, we don't try to forget this partition — one of the partitions in the session will cause an inconsistent topic ID error and the metadata for this partition will be refreshed — this should result in the old ID being removed from the session. This should not happen if the FetchSessionHandler is correctly in sync.
 * For the forgotten partitions we have the same cases: 1. partition in session is resolved, forgotten is resolved: trivial
 2. partition in session is unresolved, forgotten is unresolved: in code, this is equivalent to the case above, so trivial as well
 3. partition in session is unresolved, forgotten is resolved: this means the partition in the session does not have a name, but the metadata cache now contains the name – to fix this we can check if there exists a cached partition with the given ID and try to forget it before we check the resolved name case.
 4. partition in session is resolved, update is unresolved: this means the partition in the session has a name, but the update was unable to be resolved (ie, the topic is deleted) We will look up the partition using the ID. We will find the old version with a name and be able to delete it.
 5. both partitions in the session have IDs, but they are different: This should be the same case as described above. If we somehow do not have the ID in the session, no partition will be removed. This should not happen unless the Fetch Session Handler is out of sync.
 * Partitions in the session - there may be some partitions in the session already that are unresolved. We can resolve them in forEachPartition using a method that checks if the partition is unresolved and tries to resolve it using a topicName map from the request. The partition will be resolved before the function using the cached partition is applied. Reviewers: David Jacot <djacot@confluent.io>

view details

push time in 6 days

Pull request review commentconfluentinc/ksql

chore: add key/value schema name prop and refactor

+/*+ * Copyright 2021 Confluent Inc.+ *+ * Licensed under the Confluent Community License (the "License"); you may not use+ * this file except in compliance with the License.  You may obtain a copy of the+ * License at+ *+ * http://www.confluent.io/confluent-community-license+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+ * WARRANTIES OF ANY KIND, either express or implied.  See the License for the+ * specific language governing permissions and limitations under the License.+ */++package io.confluent.ksql.serde.json;++import static io.confluent.ksql.serde.connect.ConnectProperties.FULL_SCHEMA_NAME;+import static org.hamcrest.MatcherAssert.assertThat;+import static org.hamcrest.Matchers.is;+import static org.hamcrest.Matchers.nullValue;+import static org.junit.Assert.assertThrows;++import com.google.common.collect.ImmutableMap;+import com.google.common.collect.ImmutableSet;+import io.confluent.ksql.serde.connect.ConnectProperties;+import io.confluent.ksql.util.KsqlException;+import org.junit.Test;++public class JsonPropertiesTest {++  @Test+  public void shouldGetSupportedProperties() {+    // Given:+    final JsonProperties properties = new JsonProperties(ImmutableMap.of());++    // When:+    final ImmutableSet<String> supportedProperties = properties.getSupportedProperties();++    // Then:+    assertThat(supportedProperties, is(JsonProperties.SUPPORTED_PROPERTIES));+  }++  @Test+  public void shouldGetFullSchemaName() {+    // Given:+    final JsonProperties properties = new JsonProperties(ImmutableMap.of());++    // When:+    final Exception e = assertThrows(UnsupportedOperationException.class,+        properties::getFullSchemaName);++    // Then:+    assertThat(e.getMessage(), is("JSON does not implement Schema Registry support"));+  }++  @Test+  public void shouldGetDefaultFullSchemaName() {

Seem this test is a duplicate of the previous one?

lihaosky

comment created time in 7 days

Pull request review commentconfluentinc/ksql

chore: add key/value schema name prop and refactor

+/*+ * Copyright 2021 Confluent Inc.+ *+ * Licensed under the Confluent Community License (the "License"); you may not use+ * this file except in compliance with the License.  You may obtain a copy of the+ * License at+ *+ * http://www.confluent.io/confluent-community-license+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT+ * WARRANTIES OF ANY KIND, either express or implied.  See the License for the+ * specific language governing permissions and limitations under the License.+ */++package io.confluent.ksql.serde.json;++import com.google.common.collect.ImmutableSet;+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;+import io.confluent.ksql.serde.connect.ConnectProperties;+import java.util.Map;++public class JsonProperties extends ConnectProperties {++  static final ImmutableSet<String> SUPPORTED_PROPERTIES = ImmutableSet.of();

Why do we need to introduce this class if the set of supported properties is empty?

lihaosky

comment created time in 7 days

Pull request review commentconfluentinc/ksql

chore: add key/value schema name prop and refactor

 public void shouldNotSetAvroSchemaNameForNonAvroKey() {   public void shouldSetValidAvroSchemaName() {     // When:     final CreateSourceAsProperties properties = CreateSourceAsProperties.from(-        ImmutableMap.of(CommonCreateConfigs.VALUE_AVRO_SCHEMA_FULL_NAME, new StringLiteral("schema")));+        ImmutableMap.of(VALUE_AVRO_SCHEMA_FULL_NAME, new StringLiteral("schema")));      // Then:-    assertThat(properties.getValueFormatProperties().get(AvroFormat.FULL_SCHEMA_NAME), is("schema"));+    assertThat(properties.getValueFormatProperties().get(ConnectProperties.FULL_SCHEMA_NAME), is("schema"));+  }++  @Test+  public void shouldSetValueFullSchemaName() {+    // When:+    final CreateSourceAsProperties properties = CreateSourceAsProperties.from(+        ImmutableMap.<String, Literal>builder()+            .put(CommonCreateConfigs.VALUE_FORMAT_PROPERTY, new StringLiteral("Protobuf"))+            .put(CommonCreateConfigs.VALUE_SCHEMA_FULL_NAME, new StringLiteral("schema"))+            .build());++    // Then:+    assertThat(properties.getValueFormatProperties().get(ConnectProperties.FULL_SCHEMA_NAME), is("schema"));+  }++  @Test+  public void shouldSetFullKeySchemaName() {+    // Given:+    final CreateSourceAsProperties props = CreateSourceAsProperties+        .from(ImmutableMap.<String, Literal>builder()+            .put(FORMAT_PROPERTY, new StringLiteral("Json_sr"))+            .put(KEY_SCHEMA_FULL_NAME, new StringLiteral("KeySchema"))+            .build());++    // When / Then:+    assertThat(props.getKeyFormatProperties("json_sr", "foo"),+        hasEntry(ConnectProperties.FULL_SCHEMA_NAME, "KeySchema"));

nit: why hasEntry compared to ...get(...), is(...) as above?

lihaosky

comment created time in 7 days

more