如何使用python中的结构化火花流使用ForeachWriter将行插入到Mongodb中？

Question

I'm using spark streaming to read data from kafka and insert that into mongodb. I'm using pyspark 2.4.4. I'm trying to make use of ForeachWriter because just using for each method means the connection will establishing for every row.

    def open(self, partition_id, epoch_id):
        # Open connection. This method is optional in Python.
        self.connection = MongoClient("192.168.0.100:27017")
        self.db = self.connection['test']
        self.coll = self.db['output']    
        print(epoch_id)
        pass

    def process(self, row):
        # Write row to connection. This method is NOT optional in Python.
        #self.coll=None -> used this to test, if I'm getting an exception if it is there but I'm not getting one
        self.coll.insert_one(row.asDict())
        pass

    def close(self, error):
        # Close the connection. This method in optional in Python.
        print(error)
        pass

df_w=df7\
        .writeStream\
        .foreach(ForeachWriter())\
        .trigger(processingTime='1 seconds') \
        .outputMode("update") \
        .option("truncate", "false")\
        .start()df_w=df7\
        .writeStream\
        .foreach(ForeachWriter())\
        .trigger(processingTime='1 seconds') \
        .outputMode("update") \
        .option("truncate", "false")\
        .start()

My problem it's not inserting to mongodb and I can't find solutions for this. If comment it out I'll get error. But process method is not executing. any one have any ideas?

Answer 1

You set up the collection to None in the first line of the process function. Therefore you insert the row into nowhere. Also, I don't know if it just here, or in your code as well, but you have the writeStream part two times.

Answer 2

This is probably not documented in spark docs. But if you look at the definition of foreach in pyspark, it has the following line of code:

# Check if the data should be processed
  should_process = True
  if open_exists:
    should_process = f.open(partition_id, epoch_id)

Therefore, whenever we open a new connection, the open must return True. In actual documentation, they have used 'pass' which results in 'process()' never getting called. (This answer is for future reference for anybody facing the same issue.)