简体   繁体   中英

How to convert RDD into Dataframe with Pyspark?

I have an RDD below, which I have received from a client. How can I convert this RDD into a Dataframe?

["Row(Moid=2, Tripid='11', Tstart='2007-05-28 08:53:14.040', Tend='2007-05-28 08:53:16.040', Xstart='9738.73', Ystart='103.246', Xend='9743.73', Yend='114.553')"]

Note: This is not really an answer, but I don't understand as to what OP is asking about. Writing this in comment section would not have been possible, but may be we can take it forward from here.

OP says that he/she receives an RDD (purportedly a single element) from Client -

["Row(Moid=2, Tripid='11', Tstart='2007-05-28 08:53:14.040', Tend='2007-05-28 08:53:16.040', Xstart='9738.73', Ystart='103.246', Xend='9743.73', Yend='114.553')"]

Now, OP wants to translate that to a DataFrame. To translate that, one has to de-string the Row object, but OP has to clarify what he needs.

from pyspark.sql import Row
rdd_from_client = [Row(Moid=2, Tripid='11', Tstart='2007-05-28 08:53:14.040', Tend='2007-05-28 08:53:16.040', Xstart='9738.73', Ystart='103.246', Xend='9743.73', Yend='114.553')]
df = sqlContext.createDataFrame(rdd_from_client)
df.show(truncate=False)
+----+-----------------------+------+-----------------------+-------+-------+-------+-------+
|Moid|Tend                   |Tripid|Tstart                 |Xend   |Xstart |Yend   |Ystart |
+----+-----------------------+------+-----------------------+-------+-------+-------+-------+
|2   |2007-05-28 08:53:16.040|11    |2007-05-28 08:53:14.040|9743.73|9738.73|114.553|103.246|
+----+-----------------------+------+-----------------------+-------+-------+-------+-------+

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM