Can we use Kstream with Spark Streaming in 2021?

Question

Can we use Kstream with Spark in 2021?

Is it a recommended approach or using the Spark Streaming is a better solution.

Answer 1

You cannot. Kafka Streams and Spark (Streaming) are not compatible to be used together.

You can, alternatively, use Kubernetes, for example as a Spark Scheduler together with containerized Kafka Streams deployments that use the same input and output topics.

Answer 2

Can we use Kstream with Spark in 2021?

Sure.

Is it a recommended approach

Not really (if at all).

using the Spark Streaming is a better solution.

"Better" begs for another question "In what scenarios?"

---

Kafka Streams is a library and as such can be used anywhere a library could be used in an application, incl. Spark applications. In that sense, it is possible, but not really of much help IMHO.

Kafka Streams is simply a bunch of threads that use Consumer and Producer APIs to transform records. That's the Kafka data source in Spark Structured Streaming exactly.

Kafka Streams applications are deployed as standalone Java applications (eg Docker containers in k8s). The same is also possible with Spark Structured Streaming (with Spark on Kubernetes).

See no real benefit using both in a single application but would love proven wrong.