apache spark Streaming textFileStream - reading gzip files

Question

I am processing files placed in HDFS using Spark streaming. Specifically using textFileStream method of the JavaStreamingContext class.

As the method name contains 'text' I assumed that this will only read text files, but to my surprise it is also reading gzipped text files.

Can anyone please clarify if this is the expected behavior and what all formats can it read?

Answer 1

Yes, Spark uses Hadoop's File I/O API, which handles compression formats transparently. Even for output, you can configure the compression that should be used through a property setting and the API will handle it.