Spark CSV与各种分隔符到DataSet

Question

I have two CSV files that I am aggregating using spark with Java. These files have different delimeters.

file1.dat:

011!345!Ireland

files2.dat:

022Ç486ÇBrazil

Code I use:

Dataset<Row> people = spark.read().format("csv").option("header", "false").option("delimeter", "\u00C7").option("delimeter", "!").load(logFile);

Output:

Error:Cannot resolve column name

If I remove one delimeter:

Dataset<Row> people = spark.read().format("csv").option("header", "false").option("delimeter", "\u00C7").load(logFile);

Output:

Sno|code|Country
null|null|null 
022|486|Brazil

Is there a way to perform this? Can both of these files be aggregated in the same spark code?

Answer 1

You can't use multiple delimiters at the same time.

Instead, read the files separatly and use union ( doc ) to merge them together. For example:

Dataset<Row> people1 = spark.read()
  .option("header", "false")
  .option("delimeter", "!")
  .csv(logFile1);
Dataset<Row> people2 = spark.read()
  .option("header", "false")
  .option("delimeter", "\u00C7")
  .csv(logFile2);

Dataset<Row> people = people1.union(people2);