Why KSQL query returns null values from the stream created by stream-stream join?

Question

I'm facing an unexpected behaviour on stream-stream join query result.

Situation

KSQL Version: 5.1.3

1. There are 2 streams created from each kafka topic

#1.
CREATE STREAM streamA
(id VARCHAR)
WITH (KAFKA_TOPIC='topicA', VALUE_FORMAT='JSON');

#2.
CREATE STREAM streamB
(id VARCHAR,
date VARCHAR,
count INT)
WITH (KAFKA_TOPIC='topicB', VALUE_FORMAT='JSON')

2. stream-stream join with creating kafka topic

CREATE STREAM streamC
WITH (KAFKA_TOPIC='topicC', VALUE_FORMAT='JSON', PARTITIONS=5) AS
SELECT b.id AS `id`,
    b.date AS `date`,
    b.count AS `count`
FROM streamB b
INNER JOIN streamA a WITHIN 1 DAY
    on b.id = a.id;

Question

In this situation, when I make following 2 queries, one can get all info, but the other can't. Do you have some ideas why it happens or some problems on my queries?

Reference

Following queries returns expected result.

# OK
ksql> select a.id as `id`, a.date as `date`, a.count as `count` from streamA a inner join streamB b within 1 day on a.id = b.id;
# 00000001 | 2020-06-22 | 3

# OK
ksql> print 'topicC' from beginning;
# {"ROWTIME":1592804456184,"ROWKEY":"00000001","date":"2020-06-22","count":3}

However, following query returns unexpected one.

# NG
ksql> select * from streamC;
# 1592804456184 | 00000001 | null | null

# (expected result)
# 1592804456184 | 00000001 | 2020-06-22 | 3

Additional info

ksql> DESCRIBE streamC;
Name                 : STREAMC
 Field   | Type
-------------------------------------
 ROWTIME | BIGINT           (system)
 ROWKEY  | VARCHAR(STRING)  (system)
 date    | VARCHAR(STRING)
 count   | INTEGER
-------------------------------------
For runtime statistics and query details run: DESCRIBE EXTENDED <Stream,Table>;

Answer 1

You're running the version of KSQL that comes with Confluent Platform version 5.1.3, and it looks like you've run into a bug. I've not found an issue covering your exact issue. However, the latest releases do have test cases covering similar usage patterns so I'm confident your issue has already been fixed. All you need to do is upgrade to a version containing the fix.

I'd recommend upgrading to 5.5.1 CP release, the upcoming 6.0.0 CP release, or the community 0.10 ksqlDB release, as I know these contain the fix.

Answer 2

I am facing the same issue with Stream-Table Joins. My Table key is exactly the same as the Stream key, however the Join is not working.

I am running the last version of confluent 5.5.1.