如何将 spark Streaming dataframe 列转换为 Python 列表

Question

I have a spark streaming dataframe like below. I want to convert

+----------------+-------+-----------+-----------------+
|application_name|     id|syntheticid|          Journey|
+----------------+-------+-----------+-----------------+
|            test|   24  |   12392234|      Activation |
|            test|   24  |   12392234|          LOAD   |
+----------------+-------+-----------+-----------------+

1. How do I convert this to a normal dataframe?

2. How to convert the streaming dataframe column to a list? For example I want to convert the column journey into python list ['Activation','Load'] .

Any help would be appreciated.

Answer 1

With the limited description that you provided, one option could be saving the file first as csv with pipe | as delimiter and then reading it back using pandas.read_csv(). Suppose that you have saved file as "streaming.csv". Then you can get your python list as,

    df = pd.read_csv("streaming.csv", sep="|")
    journey = df['Journey'].tolist()

Consider selecting this as "working solution" if it solves the purpose.

Answer 2

In terms of your first point, you're not asking the correct question. Since Spark 2.0, the APIs mostly overlap, therefore a Spark Streaming DataFrame is essentially the same thing as a Spark (SQL) DataFrame, albeit Spark Streaming DataFrame is unbounded.

Since Spark 2.0, DataFrames and Datasets can represent static, bounded data, as well as streaming, unbounded data.

Therefore, you should be able to perform majority of your necessary manipulations on your (streaming) DataFrame.

In terms of your second point, try to have a look at aggregation functions such as collect_list() and collect_set() . Try this code:

from pyspark.sql import SparkSession
from pyspark.sql import functions as f

>>> spark = SparkSession.builder.getOrCreate()
>>> df = spark._sc.parallelize([
 ["test","24","12392234","Activation"], 
 ["test","24","12392234","Load"]]
).toDF(["application_name","id","syntheticid","journey"])

>>> df.show()
+----------------+---+-----------+----------+
|application_name| id|syntheticid|   journey|
+----------------+---+-----------+----------+
|            test| 24|   12392234|Activation|
|            test| 24|   12392234|      Load|
+----------------+---+-----------+----------+


>>> grouped_df = df.groupBy('application_name').agg(f.collect_list('journey').alias('collection'))
>>> grouped_df.show()
+----------------+------------------+
|application_name|        collection|
+----------------+------------------+
|            test|[Activation, Load]|
+----------------+------------------+


>>> python_list = [item for sublist in [row.collection for row in grouped_df.collect()] for item in sublist]

>>> python_list
['Activation', 'Load']