How to use OutputStreams with chunked response in Play 2.1

Question

I need to use to java.util.zip.ZipOutputStream to respond with a compressed file archive.

The data is several hundred megabytes uncompressed, so I would like to store as little of it as possible. It is coming from a serialization of SQL results.

I see examples of using an OutputStream to return a chunked result using Enumerator.outputStream :

• http://greweb.me/2012/11/play-framework-enumerator-outputstream/
• Play/Akka integration with Java OutputStreams

but those seem ill-advised when I read the documentation (emphasis mine)

Create an Enumerator of bytes with an OutputStream.

Not that calls to write will not block, so if the iteratee that is being fed to is slow to consume the input, the OutputStream will not push back. This means it should not be used with large streams since there is a risk of running out of memory .

Clearly, I can't use that. Or at least not without modification.

---

How can I create a response with an OutputStream (in this case, a gzipped archive) while being assured that only portions of it will be stored in memory?

I recognize the difference between InputStream s/ OutputStream s and Play's Enumerator / Iteratee paradigm, so I expect there will be a specific way in which I need to generate my source data (serialization of SQL results) so that it doesn't outpace the rate of download. I don't know what it is.

Answer 1

In general you can't safely use any OutputStream with the Enumerator/Iteratee framework because OutputStream doesn't support non-blocking pushback. However, if you can control the writing to the OutputStream you can hack together something like:

val baos = new ByteArrayOutputStream
val zos = new ZipOutputStream(baos)

val enumerator = Enumerator.generateM {
  Future.successful {
    if (moreDateToWrite) {
      // Write data into zos
      val r = Some(baos.toByteArray)
      baos.reset()
      r
    } else None
  }
}

If all you need is compression, take a look at the Enumeratee instances provided in play.filters.gzip.Gzip and the play.filters.gzip.GzipFilter filter.

Answer 2

The only backpressure mechanism for OutputStream is blocking the thread. So one way or another, there will have to be a thread that is able to be blocked.

One way is to use piped streams.

import java.io.OutputStream
import java.io.PipedInputStream
import java.io.PipedOutputStream
import play.api.libs.iteratee.Enumerator
import scala.concurrent.ExecutorContext

def outputStream2(a: OutputStream => Unit, bufferSize: Int)
    (implicit ec1: ExecutionContext, ec2: ExecutionContext) = {
  val outputStream = new PipedOutputStream
  Future(a(outputStream))(ec1)
  val inputStream = new PipedInputStream(pipedOutputStream, bufferSize)
  Enumerator.fromStream(inputStream)(ec2)
}

Since the operations are blocking, you must take care to prevent deadlock.

Either use two different thread pools, or used a cached (unbounded) thread pool.