How to transform a nested array into a table in ksqldb/kafka

Question

I started learning ksqldb for Kafka and ran into a problem, I have a Product stream and the structure is the following:

 ID          | VARCHAR(STRING)
 NAME        | VARCHAR(STRING)
 ...
 INGREDIENTS | ARRAY<STRUCT<NAME VARCHAR(STRING), ISALLERGEN BOOLEAN, RELATIVEAMOUNT DOUBLE>>

And as a result, I want to create a table and get the last state of the product using 'LATEST_BY_OFFSET' function, but the problem is that I can't apply this function to Array or Struct.

Should I EXPLODE each prop and create a separate entry in the table for each ingredient?(I think it's very strange). How can I deal with this situation? Can you tell me if you have any ideas?

Answer 1

This is a known issue. It is documented here: https://github.com/confluentinc/ksql/issues/5437 and I happen to be working on it presently. Hopefully a solution will be available in an upcoming version!

I'm not sure how your workaround would work given that after you explode the array that there could be several instance of each component of the struct per key...

Answer 2

The problem was solved in release 0.25.1 https://github.com/confluentinc/ksql/pull/8878 https://github.com/confluentinc/ksql/commit/adac45855dce1c413073e5cbeb474cb022013a2b