类型错误：字段 col1：LongType 不能接受类型中的对象“”<class 'str'>

Question

I have json in python like this:

example = [{"col1":"","col2":"","col3":52272}, ...]

Columns of json might be null. Empty value is "".

I created the spark schema:

schema = StructType([
   StructField("col1", LongType(), True),
   StructField("col2", LongType(), True),
   StructField("col3", LongType(), True),]

I try to get the spark dataframe like this:

pandas_df = pd.DataFrame(example)
spark_df = spark.createDataFrame(pandas_df, schema = schema)

But I get that error:

TypeError: field col1: LongType can not accept object '' in type <class 'str'>

What fix the error? Same error happens if I used other types of this column.

Answer 1

As @tdelaney commented, your schema doesn't reflect your data.

You could try something like this:

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from pyspark.sql.types import IntegerType, StringType, StructField, StructType

if __name__ == "__main__":
    spark = SparkSession.builder.master("local").appName("Test").getOrCreate()
    data = [{"col1": "", "col2": "", "col3": 52272}]
    schema = StructType(
        [
            StructField("col1", StringType(), True),
            StructField("col2", StringType(), True),
            StructField("col3", IntegerType(), True),
        ]
    )
    df = spark.createDataFrame(data=data, schema=schema)

Which gives:

+----+----+-----+
|col1|col2|col3 |
+----+----+-----+
|    |    |52272|
+----+----+-----+

If, for example, you want to replace empty strings with None you could use:

df = df.withColumn("col2", F.when(F.col("col2") != "", F.col("col2")).otherwise(None))

Which gives:

+----+----+-----+
|col1|col2|col3 |
+----+----+-----+
|    |null|52272|
+----+----+-----+