Spark Structured Streaming performance for Scala vs Python

Question

Hi~ I'm going to develop a mini-batched program with Kafka + Spark Structured Streaming . But I am very confused, whether to use python or scala, which is faster. It would be better if there is any benchmark performance result about Spark Structured Streaming between Scala and Python.

Answer 1

Not really an issue.

Only thing are that 1) Scala is faster but the scale of data per microbatch may mean less of an impact and 2) Scala has dataset support for types, pyspark does not.

Most use Scala, pyspark more for data science.

That said real-time machine learning may well be better with pyspark. See for example: https://towardsdatascience.com/building-a-real-time-prediction-pipeline-using-spark-structured-streaming-and-microservices-626dc20899eb