简体   繁体   中英

How to convert Pyspark dataframe to Python Dictionary

I'm new to pyspark have a requirement like below

A dataframe having two columns with (id and data_list) with data_list sorted order after group by like below

+---+-----+-----+
| id| data|value|
+---+-----+-----+
|1_a|AB,Ca|   10|
|1_a|Cd,da|    5|
|1_a|aC,BE|   15|
|1_a|ER,rK|   20|
|2_b|JK,Lh| 1500|
|2_b|Yu,HK|  500|
|2_b|MK,HN|  100|
+---+-----+-----+

after sorted data_list

+---+--------------------+
| id|           data_list|
+---+--------------------+
|1_a|[Cd,da, AB,Ca, aC...|
|2_b|[MK,HN, Yu,HK, JK...|
+---+--------------------+

applying map tranformation on DF to get my desired(python dictionary of list) output,

data = order_df.rdd.map(lambda (x, y): (x.split("_")[1].lower(), (x.split("_")[0].lower(), y))) \
    .groupByKey().mapValues(list)

output

  [('b', [('2', '[MK,HN, Yu,HK, JK,Lh]')]), ('a', [('1', '[Cd,da, AB,Ca, aC,BE, ER,rK]')])] 

then iterating list to get each element as below

for dd in data.collect():
    print "==========", dd[1][0][1]
    for r in dd[1][0][1]:
        print r + "---"

Desired output

Cd,da
AB,Ca
aC,BE
ER,rK
....

but getting as below

   ========== [Cd,da, AB,Ca, aC,BE, ER,rK]
ttttt:  <type 'str'>
[
C
d
,
d
a
,

A
B
,
C
a
,

a
C
,
B
E
,

E
R
,
r
K
]

Below is the code the trying to get output.

 from pyspark import SparkContext, SparkConf
        from pyspark.sql import SQLContext
        from pyspark.sql.types import *
        from pyspark.sql import functions as F
        import operator

        conf = SparkConf().setMaster("local").setAppName("Demo DF")
        sc = SparkContext(conf=conf)
        sqlContext = SQLContext(sparkContext=sc)
        sqlContext.setConf("spark.sql.shuffle.partitions", "3")

       def foo((x, y)):
          z = x.lower().split('_')
          return (z[1], (z[0], ast.literal_eval(json.dumps(y, 
               ensure_ascii=False).encode('utf8'))))

        # define udf
        def sorter(l):
            res = sorted(l, key=operator.itemgetter(1))
            return [item[0] for item in res]


        sort_udf = F.udf(sorter)

       ll_list = [("1_a", "AB,Ca", 10), ("1_a", "Cd,da", 5), ("1_a", "aC,BE", 15), ("1_a", "ER,rK", 20),
               ("2_b", "JK,Lh", 1500), ("2_b", "Yu,HK", 500), ("2_b", "MK,HN", 100)]
    input_df = sc.parallelize(ll_list).toDF(["id", "data", "value"])
        input_df.show()

        # create list column
        grouped_df = input_df.groupby("id") \
            .agg(F.collect_list(F.struct("data", "value")) \
                 .alias("list_col"))

        # test
        order_df = grouped_df.select("id", sort_udf("list_col") \
                                     .alias("data_list"))
        order_df.show()
        data = order_df.rdd.map(foo).groupByKey().mapValues(list)

        for dd in data.collect():
            print "==========", dd[1][0][1]
            for r in dd[1][0][1]:
                print r + "---"

Could you please anyone help me with this code to get the correct output.

The issue is that "data_list" is actually a column of strings:

order_df.dtypes
# [('id', 'string'), ('data_list', 'string')]

You can use ast.literal_eval to parse them.

import ast 

def foo((x, y)): 
    z = x.lower().split('_')
    return (z[1], (z[0], ast.literal_eval(y)))

order_df.rdd.map(foo).groupByKey().mapValues(list).collect()

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM