为每个文件创建一个包含架构数据的数据框

Question

I am trying to create a dataframe, then run a for loop which looks at a bunch of files. Runs through each one and addeds a row to the dataframe for file. Containing the file name and the schema details?

# Schema    
schema = StructType([
    StructField("filename", StringType(), True),
    StructField("converteddate", StringType(), True),
    StructField("eventdate", StringType(), True)
])


# Create empty dataframe
df = spark.createDataFrame(sc.emptyRDD(), schema)


for files in mvv_list:
    loadName = files
    videoData = spark.read\
                     .format('parquet')\
                     .options(header='true', inferSchema='true')\
                     .load(loadName)
    dataTypeList = videoData.dtypes
    two = dataTypeList[:2]
    print(loadName)
    print(two)

#mnt/master-video/year=2018/month=03/day=24/part-00004-tid-28948428924977-e0fc2-c85b-4296-8a05-94c5af6-2427-c000.snappy.parquet
#[('converteddate', 'timestamp'), ('eventdate', 'timestamp')]

#mnt/master-video/year=2017/month=05/day=12/part-00004-tid-2894842977-e0f21c2-c85b-4296-8a05-94c5af6-2427-c000.snappy.parquet
#[('converteddate', 'timestamp'), ('eventdate', 'date')]

#mnt/master-video/year=2016/month=03/day=24/part-00004-tid-2884924977-e0f2512-c8b-4296-8a05-945a6-2427-c000.snappy.parquet
#[('converteddate', 'timestamp'), ('eventdate', 'string')]

I am struggling to create a row and append it to the dataframe.

Wanted output

+-----------------------------+-----------------+---------------------+
|filename                     |converteddate    |eventdate            |
+-----------------------------+-----------------+---------------------+
|mnt/master-video/year=2018...|timestamp        |timestamp            |
|mnt/master-video/year=2017...|timestamp        |date                 |
|mnt/master-video/year=2016...|timestamp        |string               |
+-----------------------------+-----------------+---------------------+

Answer 1

One way is to build your desired data as a list, and then create the DataFrame after (instead of trying to append rows)

data = []
for files in mvv_list:
    loadName = files
    videoData = spark.read\
                     .format('parquet')\
                     .options(header='true', inferSchema='true')\
                     .load(loadName)
    dataTypeDict = dict(videoData.dtypes)
    data.append((loadName, dataTypeDict['converteddate'], dataTypeDict['eventdate']))

schema = StructType([
    StructField("filename", StringType(), True),
    StructField("converteddate", StringType(), True),
    StructField("eventdate", StringType(), True)
])

df = spark.createDataFrame(data, schema)