通过查询mysql来发火花rdd

Question

I use spark streaming to stream data from Kafka and I want to filter data judge by data in MySql.

For example, I get data from kafka just like:

{"id":1, "data":"abcdefg"}

and there are data in MySql like this:

id  | state  
1   | "success"

I need to query the MySql to get the state of term id. I can define a connect to MySql in the function of filter, and it works. The code like this:

def isSuccess(x):
    id = x["id"]
    sql = """
        SELECT * 
        FROM Test
        WHERE id = "{0}"
        """.format(id)
    conn = mysql_connection(......)
    result = rdbi.query_one(sql)
    if result == None:
        return False
    else:
        return True
successRDD = rdd.filter(isSuccess)

But it will define connection for every row of the RDD, and will waste a lot of computing resource.

How to do in filter?

Answer 1

I suggest you go for using mapPartition available in Apache Spark to prevent initialization of MySQL connection for every RDD.

This is the MySQL table that I created:

create table test2(id varchar(10), state varchar(10));

With the following values:

+------+---------+
| id   | state   |
+------+---------+
| 1    | success |
| 2    | stopped |
+------+---------+

Use the following PySpark Code as reference:

import MySQLdb

data1=[["1", "afdasds"],["2","dfsdfada"],["3","dsfdsf"]] #sampe data, in your case streaming data
rdd = sc.parallelize(data1)

def func1(data1):
    con = MySQLdb.connect(host="127.0.0.1", user="root", passwd="yourpassword", db="yourdb")
    c=con.cursor()
    c.execute("select * from test2;")
    data=c.fetchall()
    dict={}
    for x in data:
        dict[x[0]]=x[1]
    list1=[]
    for x in data1:
        if x[0] in dict:
            list1.append([x[0], x[1], dict[x[0]]])
        else:
            list1.append([x[0], x[1], "none"]) # i assign none if id in table and one received from streaming dont match
    return iter(list1)

print rdd.mapPartitions(func1).filter(lambda x: "none" not in x[2]).collect()

The output that i got was:

[['1', 'afdasds', 'success'], ['2', 'dfsdfada', 'stopped']]