Java or Scala libraries for stream parsers

Question

As far as I understand Scala parser combinators require the whole file to be in memory and it's not quite practical for files tens of gigabytes in size. Anyway, parser combinator library is way more powerful than needed for task at hand so I'm thinking about rolling out simple FSA parser. Are there some libraries for this or there's no other way except wall of text with multiple switch es?

Answer 1

As far as I understand Scala parser combinators require the whole file to be in memory and it's not quite practical for files tens of gigabytes in size

If you need to parse a huge file, you can use a BufferedReader and invoke parseAll with that reader. Something like this:

val bufferedReader: BufferedReader = ???
val myResult = MyParser.parseAll(MyParser.rootParser, bufferedReader)

This way the file is read in chunks so that you shouldn't be bothered to run out of memory.

Alternatively, you could create a PagedSeqReader from a bufferedSource if you need a reader specific to the parser combinators API.

val rdr = new PagedSeqReader(PagedSeq.fromReader(Source.fromFile(new File("huge_file")).bufferedReader()))