如何使用spark python在cassandra表中保存数据？

Question

I'm trying to create consumer-producer application.

Application's Producer will produce some data on particular topic. Consumer will consume this data from the same topic and process it using spark api and store this data is cassandra table.

Incoming data comming in string format like below -

100=NO|101=III|102=0.0771387731911|103=-0.7076915761 100=NO|101=AAA|102=0.8961325446464|103=-0.5465463154

I created consumer in the bellow manner:

from kafka import KafkaConsumer
from StringIO import StringIO
import pandas as pd
from cassandra.cluster import Cluster

from pyspark import SparkConf, SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

def main():

    sc = SparkContext(appName="StreamingContext")
    ssc = StreamingContext(sc, 3)

    kafka_stream = KafkaUtils.createStream(ssc, "localhost:2181", "sample-kafka-app", {"NO-topic": 1})
    raw = kafka_stream.flatMap(lambda kafkaS: [kafkaS])
    clean = raw.map(lambda xs: xs[1].split("|"))
    my_row = clean.map(lambda x: {
       "pk": "uuid()",
       "a": x[0],
       "b": x[1],
       "c": x[2],
       "d": x[3],
    })

    my_row.saveToCassandra("users", "data")
    stream.start()
    stream.awaitTermination()

if __name__ == "__main__":
    main()

Cassandra table structure -

cqlsh:users> select * from data;

 pk | a | b | c | d
----+---+---+---+---
CREATE TABLE users.data (
    pk uuid PRIMARY KEY,
    a text,
    b text,
    c text,
    d text
)

I'm facing below error -

Traceback (most recent call last):


File "consumer_no.py", line 84, in <module>
    main()
  File "consumer_no.py", line 53, in main
    my_row.saveToCassandra("users", "data")
AttributeError: 'TransformedDStream' object has no attribute 'saveToCassandra'
17/04/04 14:29:22 INFO SparkContext: Invoking stop() from shutdown hook

Is I'm going on a correct way to achieve which I explain above? If not then give me suggestions to achieve this and if yes then what's wrong/missing in above code?

Answer 1

Rather than directly trying to save your TransformedDStream to Cassandra, you should save each RDD from that DStream to cassandra.

Your code should work if you do something like this:

my_row.foreachRDD(lambda x: x.saveToCassandra("users", "data"))