兑换 <class 'pyspark.sql.types.Row'> 对象到数据框-pyspark
[英]convert <class 'pyspark.sql.types.Row'> object to dataframe - pyspark
问题描述
I want process multiple json records one after the other. 
我想一个接一个地处理多个json记录。 My code reads the multiple jsons and stores them into dataframe. 我的代码读取多个json并将它们存储到dataframe中。 Now i want to process the json document row by row from dataframe. 现在我想从数据帧中逐行处理json文档。 When i take the row from dataframe i need to convert that single row to dataframe again and do some operations on that. 当我从数据帧中获取行时，我需要再次将该行转换为数据帧，并对此进行一些操作。 I am stuck at to convert class 'pyspark.sql.types.Row' object to dataframe. 我被困在将类'pyspark.sql.types.Row'对象转换为dataframe。
df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri","mongodb://127.0.0.1/mydatabase.sample").load()
A = u(funcRowIter,df.schema)
z = df.withColumn("new_column",A(struct([df[x] for x in df.columns])))
z.show()
def funcRowIter(rows):
print type(rows)
if(rows is not None):
rdf = sqlContext.createDataFrame(rows)
rdf.show()
return rows
Help me out to convert the class 'pyspark.sql.types.Row' object to dataframe. 帮我把类'pyspark.sql.types.Row'对象转换为dataframe。 My row object is huge json file. 我的行对象是巨大的json文件。
This is the json i am trying to read from mongodb 这是我试图从mongodb读取的json
{
"Feed": {
"feedBody": {
"Reservation": {
"recordLocatorID": "X23344",
"pnrCreateDate": "2018-09-24T23:00:00.000",
"lastUpdateTimestamp": "2018-09-26T14:51:01.643",
"pnrReservationSystemSequenceID": "1643",
"pnrPurgeDate": "2018-10-11",
"passengerCount": "1",
"reservationSystemCode": "1X",
"passengerList": {
"passenger": {
"passengerID": "2",
"lastUpdateTimestamp": "2018-09-24T18:00:54.835",
"dateOfBirth": "1993-10-02",
"givenName": "fgdfg",
"surName": "fgdfg",
"gender": "M",
"infantIndicator": "true",
"seatCount": "1",
"reservationSystemCustomerID": "dfgdfg",
"passengerTypeCode": "dfgfd",
"groupDepositIndicator": "false",
"passengerTicketDocList": {
"passengerTicketDoc": {
"ticketDocID": "45",
"lastUpdateTimestamp": "2018-09-24T18:01:01.149",
"ticketNumber": "43434343434",
"ticketType": "T",
"ticketIndicator": "E",
"status": "T",
"issuanceDate": "2010-09-20",
"chargeAmount": "0.74",
"currency": "USD"
}
}
}
}
}
}
}
} }
This is the rows output 这是行输出
Row(Feed=Row(
feedBody=Row(
Reservation=Row(
recordLocatorID=u'X23344',
pnrCreateDate=u'2018-09-24T23:00:00.000',
lastUpdateTimestamp=u'2018-09-26T14:51:01.643',
pnrReservationSystemSequenceID=u'1643',
pnrPurgeDate=u'2018-10-11',
passengerCount=u'1',
reservationSystemCode=u'1X',
passengerList=Row(
passenger=Row(
passengerID=u'2',
lastUpdateTimestamp=u'2018-09-24T18:00:54.835',
dateOfBirth=u'1993-10-02',
givenName=u'fgdfg',
surName=u'fgdfg',
gender=u'M',
infantIndicator=u'true',
seatCount=u'1',
reservationSystemCustomerID=u'dfgdfg',
passengerTypeCode=u'dfgfd',
groupDepositIndicator=u'false',
passengerTicketDocList=Row(
passengerTicketDoc=Row(
ticketDocID=u'45',
lastUpdateTimestamp=u'2018-09-24T18:01:01.149',
ticketNumber=u'43434343434',
ticketType=u'T',
ticketIndicator=u'E',
status=u'T',
issuanceDate=u'2010-09-20',
chargeAmount=u'0.74',
currency=u'USD'))))))), _id=Row(oid=u'5bc0cc8c2ec34dd42a44fc2f'))
0 个解决方案
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.