How to skip footer/trailer record while loading csv file to hive table

Question

The file is CSV with comma delimited.

Framework for ingesting CSV file is present. Header from the same file is skipped by:

Df.Option(“header”, “true”)

But trailer record in the same spark package, I am unable to skip it same logic.

Please help with this data ingestion.

Answer 1

Please check this reply:

spark how to remove last line in a csv file

A copy from the same reply:

val total = df.count();
val withoutFooter = df.zipWithIndex()
                        .filter(x => x._2 < total - 3)
                        .map (x => x._1)