如何用 spark.read 方法排除第一行？

Question

I am using databricks with pyspark. I would like to read this file (with the 'spark.read' method) which has the first and last lines to be excluded:

<X> example </X>
ID
11111   
22222    
<X> example </X>

How do I exclude the first row (or multiple rows if that is the case)? I have tried using:

  df = spark.read \
    .options(header='false') 

I was unsuccessful. Alternatively adding a '#' character to the beginning of the file (because then the spark.read command can interpret it as a comment and ignore the line) but I am dealing with very large files and would like to avoid reading or unnecessary steps that lengthen the process

  line = """#"""
  with open("myfile.txt", 'r+') as file: 
    file_data = file.read() 
    file.seek(0, 0) 
    file.write(line + file_data) 
    

Answer 1

Have you tried using the option ("mode", "DROPMALFORMED"). This option will drop bad rows automatically. If the rows are still being read then try to filter them. Refer to this article as well: https://medium.com/@11amitvishwas/how-to-handle-bad-records-corrupt-records-in-apache-spark-392f2991cbb5