从csv读取数据返回空值

Question

I am trying to read data from csv using Scala and Spark but the values of columns are null.

I tried to read data from csv. I also provided a schema for querying the data easily.

private val myData= sparkSession.read.schema(createDataSchema).csv("data/myData.csv")

def createDataSchema = {
    val schema = StructType(
      Array(
        StructField("data_index",StringType, nullable = false),
        StructField("property_a",IntegerType, nullable = false),
        StructField("property_b",IntegerType, nullable = false),
        //some other columns
     )
   )

   schema

Querying data:

val myProperty= accidentData.select($"property_b")
myProperty.collect()

I expect that the data are returned as a List of certain values

but they are returned as a list containing null values (values are null). Why?

When I print the schema then nullable is set to true instead of false.

I am using Scala 2.12.9 and Spark 2.4.3.

Answer 1

While loading data from CSV file though schema has been provided as nullable = false, Still Spark overwrites schema as nullable = true, so that null pointer could be avoided during data load.

Let us take an example, let's assume CSV file has two rows with second-row has an empty or null column value.

CSV:
a,1,2
b,,2

If nullable = false, a null pointer exception would be thrown while loading data when an action has called on the data frame, as there is empty/null value to be loaded & there is no default value a Null pointer is thrown. So to avoid it Spark overwrites it as nullable = true.

However, this could be handled by replacing all null with a default value and then re-applying schema.

val df = spark.read.schema(schema).csv("data/myData.csv")
val dfWithDefault = df.withColumn("property_a", when(col("property_a").isNull, 0).otherwise(df.col("property_a")))
val dfNullableFalse = spark.sqlContext.createDataFrame(dfWithDefault.rdd, schema)
dfNullableFalse.show(10)

df.printSchema()
root
|-- data_index: string (nullable = true)
|-- property_a: integer (nullable = true)
|-- property_b: integer (nullable = true)

dfNullableFalse.printSchema()
root
|-- data_index: string (nullable = false)
|-- property_a: integer (nullable = false)
|-- property_b: integer (nullable = false)