I am currently exploring on enabling async checkpointing in Spark Structured streaming , but not able to find any way for the same. DataBricks is offe ...
I am currently exploring on enabling async checkpointing in Spark Structured streaming , but not able to find any way for the same. DataBricks is offe ...
I am working on creating a basic streaming app which reads streaming data from kafka and process the data. Below is the code I am trying in pyspark ...
I am currently exploring spark's speculative tasks option. Below are my configuration which I am planning to use. I am reading the data from kafka an ...
I am running one streaming application and processing data from Kafka to Kafka using spark. If i am using latest then its working as expected and runn ...
I am stuck with very weird issue in spark structure streaming. Whenever I am shutting down the stream and restart again it again process already proce ...
I have trying to setup the Apache Spark with kafka and wrote simple program in local and its failing and not able figure out from debug. build.gradle ...
I want to write a Spark Streaming Job from Kafka to Elasticsearch. Here I want to detect the schema dynamically while reading it from Kafka. Can you ...
I'm trying to write data pulled from a Kafka to a Bigquery table every 120 seconds. I would like to do some additional operations which by documentati ...
When we use DataStreamReader API for a format in Spark, we specify options for the format used using option/options method. For example, In the below ...
I am diving to understand how can I send(produce) a large batch of records to a Kafka Topic from Spark. From the docs I can see that there is an atte ...
I have two type of job: Spark Batch jobs and and Spark streaming jobs. I would like to schedule and manage them both with airflow. Can anyone give m ...
I have a Kafka broker with a topic connected to Spark Structured Streaming. My topic sends data to my streaming dataframe, and I'd like to get informa ...
I have the following avro schema However, when I am streaming some events via kafka to spark with this schema, the streaming data frame depicts the ...
. Answers to this question are eligible for a +50 reputation bounty. Ma ...
I am trying to count the number of words in the text and save result to the Cassandra database. Producer reads the data from the file and sends it to ...
There is no error when I submitted a jar file. But data isn't printed when I send data using the HTTP protocol. (Data is printed well when I check u ...
Is it possible to keep the streamingjob running all the time? After about 24 hours, it spits out this error and stops processing. I'm not quite sure h ...
I have a stream like I would like to use spark streaming to keep only, for each group, the most recent time. With a spark dataframe I would use a ...
We have sensors starting and running for a random duration multiple times a day. The data from the sensors are sent to a Kafka topic and is consumed b ...
What is the best way to compare received data in Spark Streaming to existing data in HBase? We receive data from kafka as DStream, and before writing ...