KSQL select delete event

Question

I have a kafka KTable, that contains user Ids. I call This Ktable "UserWhitelist". This Ktable is made from the topic "Users". If new message is inserted/updated into the Users topic, I get a new record in Ktable and event is raised. If i send a key with tombstone to the users topic, record from Ktable is removed but i dont get any event to my consumer..below is the code, how i listen to Ktable events.

using (var client = new HttpClient())
        {

            client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);

            var request = new HttpRequestMessage(HttpMethod.Post, url);
            request.Method = HttpMethod.Post;
            request.Content = new System.Net.Http.StringContent("{ \"ksql\": \"select * from UserWhitelist;\",\"streamsProperties\": { \"ksql.streams.auto.offset.reset\": \"earliest\"}}", Encoding.UTF8, "application/vnd.ksql.v1+json");
            //request.Content.Headers.Add("Accept", "application/vnd.ksql.v1+json");
            using (var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead))
            {
                using (var body = await response.Content.ReadAsStreamAsync())
                using (var reader = new StreamReader(body))
                    while (!reader.EndOfStream)
                        Console.WriteLine(reader.ReadLine());
            }
        }

Do i understand correctly, that deletes are not supposed to be raised? If so, what approach you could suggest for my situation, were i would like to have a whitelist and allways inform every consumer if any update/delete/insert was made on Ktable.

Answer 1

A KSQL select statement will always show the continuous state of the table. If you've already printed an event, then delete it, then it will still be printed and you won't see any new updates for the key until it has a value added back to the table.

In the table itself, the row will be removed, but the only way to get notified about that event, otherwise, would be to use an actual Kafka consumer on the table's underlying topic, then check for when the value of the record is null.

What you're asking for is typically done by exposing a secondary REST API via Interactive Queries from a Java KTable processor, and the KSQL ticket can be found here - https://github.com/confluentinc/ksql/issues/530

However, still, you'd be required to track all values between queries, then scan them all in order to know which were deleted, as compared to consuming all events, then filtering out non-null events