使用Spark将数据配置为Kafka主题

Question

I am trying to write data in Hive table to Kafka topic using Spark.

I am working on writing a data frame of 9 million records (per day) to a Kafka topic using the query:

val ds=df.selectExpr("topic", "CAST(key AS STRING)", "CAST(value AS STRING)")
.write.format("kafka").option("kafka.bootstrap.servers", "host1:port1,host2:port2").start()

Can this query have the capability to write that huge amount of data to the kafka topic?

If yes, how much time it could take to complete writing the data?

If not, what are the other possible ways to do it?

Answer 1

You can use batch processing if the task is to do the above mentioned operation daily and not in real-time.

9 million records can be handled easily with this.

The time required to do this depends on the cluster configuration and also on the intermediate processing required.