pyspark RDD到DataFrame

Question

I am new to Spark.

I have a DataFrame and I used the following command to group it by 'userid'

def test_groupby(df):
    return list(df)

high_volumn = self.df.filter(self.df.outmoney >= 1000).rdd.groupBy(
                    lambda row: row.userid).mapValues(test_groupby)

It gives a RDD which in following structure:

 (326033430, [Row(userid=326033430, poiid=u'114233866', _mt_datetime=u'2017-06-01 14:54:48', outmoney=1127.0, partner=2, paytype=u'157', locationcity=u'\u6f4d\u574a', locationprovince=u'\u5c71\u4e1c\u7701', location=None, dt=u'20170601')])

326033430 is the big group.

My question is how can I convert this RDD back to a DataFrame Structure? If I cannot do that, how I can get values from the Row term?

Thank you.

Answer 1

You should just

from pyspark.sql.functions import *
high_volumn = self.df\
            .filter(self.df.outmoney >= 1000)\
            .groupBy('userid').agg(collect_list('col'))

and in .agg method pass what You want to do with rest of data.

Follow this link : http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.agg