当字符串以“d”结尾时，Spark 错误地将数据类型从 csv 解释为 Double

Question

There is a CSV with a column ID (format: 8-digits & "D" at the end). When reading csv with.option("inferSchema", "true"), it returns the data type as double and trimed the "D".

ACADEMIC_YEAR_SEM	ID
2013/1	12345678D
2013/1	22345678D
2013/2	32345678D

Image: https://i.stack.imgur.com/18Nu6.png

Is there any idea (apart from inferSchema=False) to get correct result? Thanks for help!

Answer 1

You can specify the schema with .schema and pass a string with columns and their type separated by commas:

df2 = spark.read.format('csv').option("header", "true").schema("ACADEMIC_YEAR_SEM string, ID string")\
.load("pyspark_sample_data.csv")

+-----------------+--------+
|ACADEMIC_YEAR_SEM|      ID|
+-----------------+--------+
|           2013/1|1234567D|
|           2013/1|2234567D|
|           2013/2|3234567D|
+-----------------+--------+