pyspark dataframe 架构，无法为 csv 文件设置可为空的 false

Question

I am trying to load csv file using pyspark. I am giving my own schema with columns nullable false, still when I print schema it shows them true. I checked the file data, there are no null entries for columns which are nullable false.

Code

from pyspark.sql.types import *

udemy_comments_file = '/Users/harbeerkadian/Documents/workspace/learn-spark/source_data/udemy/comments_spark.csv'
schema = StructType([StructField("id",StringType(),False),
                             StructField("course_id",StringType(),True),
                             StructField("rate",DoubleType(),True),
                             StructField("date",TimestampType(),True),
                             StructField("display_name",StringType(),True),
                             StructField("comment",StringType(),True),
                    StructField("new_id",StringType(),True)])
comments_df = spark.read.format('csv').option('header', 'true').schema(schema).load(udemy_comments_file)
comments_df.printSchema()
print("non null record count for id", comments_df.filter(comments_df.id.isNull()).count())

output

root
 |-- id: string (nullable = true)
 |-- course_id: string (nullable = true)
 |-- rate: double (nullable = true)
 |-- date: timestamp (nullable = true)
 |-- display_name: string (nullable = true)
 |-- comment: string (nullable = true)
 |-- new_id: string (nullable = true)

non null record count for id 0

Ideally the id column nullable property should be false, as there are zero non null records.

Answer 1

Can you try to break the statement like below and load the data after assigning schema output to a new variable:

csv_reader = spark.read.format('csv').option('header', 'true')
comments_df = csv_reader.schema(schema).load(udemy_comments_file)
comments_df.printSchema()