pyspark 中的行数差异写入 csv

Question

I have a pyspark dataframe that I want to write to HDFS. I am using the following command: df.write.mode("overwrite").option("header", "true").option("sep", "|").csv(outfile, compression="bzip2")

I am observing a weird thing. The dataframe has 366,000 rows which I obtained using the df.count() function. However, the output of the write command only has 72, 557 lines (wc -l command). Ideally each row should have a corresponding line in the output. Is there anything wrong with the write command I have been using?

Answer 1

It turns out that there were some rows with all elements as null. And this led to the discrepancy in the line count.

And the rows were null because while reading the dataframe, I was passing a manually-defined schema. The rows that did not follow the schema got inserted as null rows in the dataframe.