Spark Streaming：如何有效地将foreachRDD数据保存到Mysql数据库中？

Question

We are going to build a real-time computation system, Also want to save processed data into Mysql Database, here's the code below:

splitWordInfo.foreachRDD(new Function<JavaRDD<String>, Void>() {
        private static final long serialVersionUID = 1L;

        @Override
        public Void call(JavaRDD<String> rdd) throws Exception {
            rdd.foreachPartition(new VoidFunction<Iterator<String>>() {
                // Default Serial ID
                private static final long serialVersionUID = 1L;
                @Override
                public void call(Iterator<String> eachline) throws Exception {
                    String sql = "insert into test_mm(name,addr) values(?)";
                    Connection conn = DriverManager.getConnection("jdbc:mysql://xx.xx.xx.xx:3306/dbname", "user", "pass");
                    PreparedStatement stat = conn.prepareStatement(sql); 
                    while(eachline.hasNext()){
                        stat.setString(1, eachline.next());
                        stat.executeUpdate();
                    }
                    stat.close();
                    conn.close();
                }

            });
            return null;
        }
    });

Does it will open/close mysql connection for each rdd, or for each partition?

And How to efficiently save foreachRDD data into Mysql database. Could anyone do me a favor?

Answer 1

Each RDD partition is like a separate task and your program will get a connection for each partition. It is good to use a connection pool library like Hikari or Tomcat . But even with connection pool there will be a cost of communication with database. That you can not avoid in this model.