Scala CSV parser removed spaces

Question

I am trying to parse the following line to an data array:

"John,Doe","123 Main St","Brown Eyes"

I wanted to have an array data like below:

data(0) = John,Doe
data(1) = 123 Main St
data(2) = Brown Eyes

I used the following CSV parser from website:

import scala.util.parsing.combinator._

object CSV extends RegexParsers {
  override protected val whiteSpace = """[ \t]""".r

  def COMMA   = ","
  def DQUOTE  = "\""
  def DQUOTE2 = "\"\"" ^^ { case _ => "\"" }
  def CR      = "\r"
  def LF      = "\n"
  def CRLF    = "\r\n"
  def TXT     = "[^\",\r\n]".r

  def file: Parser[List[List[String]]] = repsep(record, CRLF) <~ opt(CRLF)
  def record: Parser[List[String]] = rep1sep(field, COMMA)
  def field: Parser[String] = (escaped|nonescaped)
  def escaped: Parser[String] = (DQUOTE~>((TXT|COMMA|CR|LF|DQUOTE2)*)<~DQUOTE) ^^ { case ls => ls.mkString("")}
  def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") }

  def parse(s: String) = parseAll(file, s) match {
    case Success(res, _) => res
    case _ => List[List[String]]()
  }
}

Then all the spaces are trimmed. The data array actually look like:

data(0) = John,Doe
data(1) = 123MainSt
data(2) = BrownEyes

---

How do I avoid such unwanted "removing whitespace" for the CSV parser? Thanks!

Answer 1

Your code says to take a sequence of escaped or nonescaped tokens and join them with no intervening space:

...* ^^ { case ls => ls.mkString("") }

Per the docs for RegexParsers ,

• The parsing methods call the method skipWhitespace (defaults to true) and, if true, skip any whitespace before each parser is called.
• Protected val whiteSpace returns a regex that identifies whitespace.

Try turning off skipWhitespace :

override protected val skipWhitespace = false

Answer 2

Is there a particular reason for hand-writing a CSV-decoder instead of using one of many existing, well-tested ones? Like OpenCSV or Jackson CSV module . It should be much simpler to use an existing lib, and you wouldn't bump into various issues in trying to unescape quotes, trim (or not) spaces, and so on.

Answer 3

The precise answer to your question was given by Robert Starling: set skipWhitespace to false .

The answer to the question "how do I parse CSV reliably?", which I'm assuming is what you really want to know, is "use a dedicated library".

You can use one of the Java ones - opencsv, commons-csv, jackson-csv, univocity... or one of the Scala ones - product-collections, purecsv, kantan.csv...

Don't write your own without a good reason - I wrote tabulate because I needed better type handling than was available at the time - and if you do, don't use one of the Scala parser combinator libraries: they load the whole data as a string in memory before parsing, which doesn't scale at all when your data starts growing.

If you must write your own and want to use a parser combinator library (because, let's face it, it's a fun problem and those libraries are cool), consider fastparse instead, or parboiled, which are both of a higher quality than the standard Scala one.

Answer 4

You can do this job in one line :

line.split((",(?=([^\"]*\"[^\"]*\")*[^\"]*$)")

The regex is from here and then apply this on all lines of your file. This will split only the comma outside two quotes.

Code to parse the csv File :

scala> scala.io.Source.fromFile("toto.csv").getLines.toList.map(_.split((",(?=([^\\"]*\\"[^\\"]*\\")*[^\\"]*$)"))