Pyspark：如何创建只有一行的数据框？

Question

What I am trying to do seems to be quite simple. I need to create a dataframe with a single column and a single value.

I have tried a few approaches, namely:

Creation of empty dataframe and appending the data afterwards:

project_id = 'PC0000000042'
schema = T.StructType([T.StructField("ProjectId", T.StringType(), True)])
empty_df = sqlContext.createDataFrame(sc.emptyRDD(), schema)

rdd = sc.parallelize([(project_id)])
df_temp = spark.createDataFrame(rdd, SCHEMA)
df = empty_df.union(df_temp)

Creation of dataframe based on this one value.

rdd = sc.parallelize([(project_id)])
df = spark.createDataFrame(rdd, schema)

However, what I get in both cases is:

TypeError: StructType can not accept object 'PC0000000042' in type <class 'str'>

Which I don't quite understand since the type seems to be correct. Thank you for any advice!

Answer 1

One small change. If you have project_id = 'PC0000000042' , then
rdd = sc.parallelize([[project_id]])

You should pass the data as a list of list: [['PC0000000042']] instead of ['PC0000000042'] .

If you have 2 rows, then:

project_id = [['PC0000000042'], ['PC0000000043']]
rdd = sc.parallelize(project_id)
spark.createDataFrame(rdd, schema).show()

+------------+
|   ProjectId|
+------------+
|PC0000000042|
|PC0000000043|
+------------+

Without RDDs , you can also do:

project_id = [['PC0000000042']]
spark.createDataFrame(project_id,schema=schema).show()