How to stream Anorm large query results to client in chunked response with Play 2.5

Question

I have a pretty large result set (60k+ records columns) that I am pulling from a database and parsing with Anorm (though I can use play's default data access module that returns a ResultSet if needed). I need to transform and stream these results directly to the client (without holding them in a big list in memory) where they will then be downloaded directly to a file on the client's machine.

I have been referring to what is demonstrated in the Chunked Responses section in the ScalaStream 2.5.x Play documentation. I am having trouble implementing the "getDataStream" portion of what it shows there.

I've also been referencing what is demoed in the Streaming Results and Iteratee sections in the ScalaAnorm 2.5.x Play documentation. I have tried piping the results as an enumerator like what is returned here:

 val resultsEnumerator = Iteratees.from(SQL"SELECT * FROM Test", SqlParser.str("colName"))

into

val dataContent = Source.fromPublisher(Streams.enumeratorToPublisher(resultsEnumerator))
Ok.chunked(dataContent).withHeaders(("ContentType","application/x-download"),("Content-disposition","attachment; filename=myDataFile.csv"))

But the resulting file/content is empty.

And I cannot find any sample code or references on how to convert a function in the data service that returns something like this:

@annotation.tailrec
def go(c: Option[Cursor], l: List[String]): List[String] = c match {
  case Some(cursor) => {
    if (l.size == 10000000) l // custom limit, partial processing
    else {
      go(cursor.next, l :+ cursor.row[String]("VBU_NUM"))
    }
  }
  case _ => l
}

val sqlString = s"select colName FROM ${tableName} WHERE ${whereClauseStr}"

val results : Either[List[Throwable], List[String]] = SQL(sqlString).withResult(go(_, List.empty[String]))
results

into something i can pass to Ok.chunked().

So basically my question is, how should I feed each record fetch from the database into a stream that I can do a transformation on and send to the client as a chunked response that can be downloaded to a file?

I would prefer not to use Slick for this. But I can go with a solution that does not use Anorm, and just uses the play dbApi objects that returns the raw java.sql.ResultSet object and work with that.

Answer 1

After referencing the Anorm Akka Support documentation and much trial and error, I was able to achieve my desired solution. I had to add these dependencies

"com.typesafe.play" % "anorm_2.11" % "2.5.2",
"com.typesafe.play" % "anorm-akka_2.11" % "2.5.2",
"com.typesafe.akka" %% "akka-stream" % "2.4.4"

to by build.sbt file for Play 2.5.

and I implemented something like this

//...play imports
import anorm.SqlParser._
import anorm._

import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Sink, Source}

...

private implicit val akkaActorSystem = ActorSystem("MyAkkaActorSytem")
private implicit val materializer = ActorMaterializer()

def streamedAnormResultResponse() = Action {
  implicit val connection = db.getConnection()

  val parser: RowParser[...] = ...
  val sqlQuery: SqlQuery = SQL("SELECT * FROM table")

  val source: Source[Map[String, Any] = AkkaStream.source(sqlQuery, parser, ColumnAliaser.empty).alsoTo(Sink.onComplete({
    case Success(v) =>
      connection.close()
    case Failure(e) =>
      println("Info from the exception: " + e.getMessage)
      connection.close()
  }))

  Ok.chunked(source)
}