Spark：PySpark Slowness，写给Cassandra的内存问题

Question

I am using pyspark to aggregate and group a largish csv on a low end machine ; 4 GB Ram and 2 CPU Core. This is done to check the memory limits for the prototype. After aggregation I need to store the RDD to Cassandra which is running in another machine.

I am using Datastax cassandra-python driver. First I used rdd.toLocalIterator and iterated through the RDD and used the drivers synchronous API session.execute . I managed to insert about 100,000 records in 5 mts- very slow. Checking this I found as explained here python driver cpu bound , that when running nload nw monitor in the Cassandra node, the data put out by the python driver is at a very slow rate, causing the slowness

So I tried session.execute_async and I could see the NW transfer at very high speed, and insertion time was also very fast.

This would have been a happy story but for the fact that, using session.execute_async, I am now running out of memory while inserting to a few more tables (with different primary keys)

Since rdd.toLocalIterator is said to need memory equal to a partition, I shifted the write to Spark worker using rdd.foreachPartition(x) , but still going out of memory.

I am doubting that it is not the rdd iteration that causes this, but the fast serialization ? of execute_async of the python driver (using Cython)

Of course I can shift to a bigger RAM node and try; but it would be sweet to solve this problem in this node; maybe will try also multiprocessing next; but if there are better suggestions please reply

the memory error I am getting is from JVM/or OS outofmemory,

6/05/27 05:58:45 INFO MapOutputTrackerMaster: Size of output statuses for 
shuffle 0 is 183 bytes
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fdea10cc000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/ec2-user/hs_err_pid3208.log

Answer 1

I tried the execution in a machine with a bigger RAM - 16 GB; This time I am able to avert the Out if Memory scenario of above;

However this time I change the insert a bit to insert to multiple table;

So even with session.executeAsysc too I am finding that the python driver is CPU bound (and I guess due to GIL not able to make use of all CPU cores), and what goes out in the NW is a trickle.

So I am not able to attain case 2; Planning to change to Scala now

Case 1: Very less output to the NW - Write speed is fast but nothing to write

Case 2: Ideal Case - Inserts being IO bound: Cassandra writes very fast