I'm new to pyspark have a requirement like below
A dataframe having two columns with (id and data_list) with data_list sorted order after group by like below
+---+-----+-----+
| id| data|value|
+---+-----+-----+
|1_a|AB,Ca| 10|
|1_a|Cd,da| 5|
|1_a|aC,BE| 15|
|1_a|ER,rK| 20|
|2_b|JK,Lh| 1500|
|2_b|Yu,HK| 500|
|2_b|MK,HN| 100|
+---+-----+-----+
after sorted data_list
+---+--------------------+
| id| data_list|
+---+--------------------+
|1_a|[Cd,da, AB,Ca, aC...|
|2_b|[MK,HN, Yu,HK, JK...|
+---+--------------------+
applying map tranformation on DF to get my desired(python dictionary of list) output,
data = order_df.rdd.map(lambda (x, y): (x.split("_")[1].lower(), (x.split("_")[0].lower(), y))) \
.groupByKey().mapValues(list)
output
[('b', [('2', '[MK,HN, Yu,HK, JK,Lh]')]), ('a', [('1', '[Cd,da, AB,Ca, aC,BE, ER,rK]')])]
then iterating list to get each element as below
for dd in data.collect():
print "==========", dd[1][0][1]
for r in dd[1][0][1]:
print r + "---"
Cd,da
AB,Ca
aC,BE
ER,rK
....
but getting as below
========== [Cd,da, AB,Ca, aC,BE, ER,rK]
ttttt: <type 'str'>
[
C
d
,
d
a
,
A
B
,
C
a
,
a
C
,
B
E
,
E
R
,
r
K
]
Below is the code the trying to get output.
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql import functions as F
import operator
conf = SparkConf().setMaster("local").setAppName("Demo DF")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sparkContext=sc)
sqlContext.setConf("spark.sql.shuffle.partitions", "3")
def foo((x, y)):
z = x.lower().split('_')
return (z[1], (z[0], ast.literal_eval(json.dumps(y,
ensure_ascii=False).encode('utf8'))))
# define udf
def sorter(l):
res = sorted(l, key=operator.itemgetter(1))
return [item[0] for item in res]
sort_udf = F.udf(sorter)
ll_list = [("1_a", "AB,Ca", 10), ("1_a", "Cd,da", 5), ("1_a", "aC,BE", 15), ("1_a", "ER,rK", 20),
("2_b", "JK,Lh", 1500), ("2_b", "Yu,HK", 500), ("2_b", "MK,HN", 100)]
input_df = sc.parallelize(ll_list).toDF(["id", "data", "value"])
input_df.show()
# create list column
grouped_df = input_df.groupby("id") \
.agg(F.collect_list(F.struct("data", "value")) \
.alias("list_col"))
# test
order_df = grouped_df.select("id", sort_udf("list_col") \
.alias("data_list"))
order_df.show()
data = order_df.rdd.map(foo).groupByKey().mapValues(list)
for dd in data.collect():
print "==========", dd[1][0][1]
for r in dd[1][0][1]:
print r + "---"
Could you please anyone help me with this code to get the correct output.
The issue is that "data_list" is actually a column of strings:
order_df.dtypes
# [('id', 'string'), ('data_list', 'string')]
You can use ast.literal_eval
to parse them.
import ast
def foo((x, y)):
z = x.lower().split('_')
return (z[1], (z[0], ast.literal_eval(y)))
order_df.rdd.map(foo).groupByKey().mapValues(list).collect()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.