Trying to understand Scala enumerator/iteratees

Question

I am new to Scala and Play!, but have a reasonable amount of experience of building webapps with Django and Python and of programming in general.

I've been doing an exercise of my own to try to improve my understanding - simply pull some records from a database and output them as a JSON array. I'm trying to use the Enumarator/Iteratee functionality to do this.

My code follows:

TestObjectController.scala:

  def index = Action {
    db.withConnection { conn=>
      val stmt = conn.createStatement()
      val result = stmt.executeQuery("select * from datatable")

      logger.debug(result.toString)
      val resultEnum:Enumerator[TestDataObject] = Enumerator.generateM {
        logger.debug("called enumerator")
        result.next() match {
          case true =>
            val obj = TestDataObject(result.getString("name"), result.getString("object_type"),
              result.getString("quantity").toInt, result.getString("cost").toFloat)
            logger.info(obj.toJsonString)
            Future(Some(obj))
          case false =>
            logger.warn("reached end of iteration")
            stmt.close()
            null
        }
      }

      val consume:Iteratee[TestDataObject,Seq[TestDataObject]] = {
        Iteratee.fold[TestDataObject,Seq[TestDataObject]](Seq.empty[TestDataObject]) { (result,chunk) => result :+ chunk }
      }

      val newIteree = Iteratee.flatten(resultEnum(consume))
      val eventuallyResult:Future[Seq[TestDataObject]] = newIteree.run
      eventuallyResult.onSuccess { case x=> println(x)}
      Ok("")

    }
  }

TestDataObject.scala:

package models

case class TestDataObject (name: String, objtype: String, quantity: Int, cost: Float){
  def toJsonString: String = {
    val mapper = new ObjectMapper()
    mapper.registerModule(DefaultScalaModule)
    mapper.writeValueAsString(this)
  }
}

I have two main questions:

• How do i signal that the input is complete from the Enumerator callback? The documentation says "this method takes a callback function e: => Future[Option[E]] that will be called each time the iteratee this Enumerator is applied to is ready to take some input." but I am unable to pass any kind of EOF that I've found because it;s the wrong type. Wrapping it in a Future does not help, but instinctively I am not sure that's the right approach.

• How do I get the final result out of the Future to return from the controller view? My understanding is that I would effectively need to pause the main thread to wait for the subthreads to complete, but the only examples I've seen and only things i've found in the future class is the onSuccess callback - but how can I then return that from the view? Does Iteratee.run block until all input has been consumed?

A couple of sub-questions as well, to help my understanding:

• Why do I need to wrap my object in Some() when it's already in a Future? What exactly does Some() represent?
• When I run the code for the first time, I get a single record logged from logger.info and then it reports "reached end of iteration". Subsequent runs in the same session call nothing. I am closing the statement though, so why do I get no results the second time around? I was expecting it to loop indefinitely as I don't know how to signal the correct termination for the loop.

Thanks a lot in advance for any answers, I thought I was getting the hang of this but obviously not yet!

Answer 1

How do i signal that the input is complete from the Enumerator callback?

You return a Future(None) .

How do I get the final result out of the Future to return from the controller view?

You can use Action.async ( doc ):

def index = Action.async {
  db.withConnection { conn=>
    ...
    val eventuallyResult:Future[Seq[TestDataObject]] = newIteree.run
    eventuallyResult map { data =>
      OK(...)
    }
  }
}

Why do I need to wrap my object in Some() when it's already in a Future? What exactly does Some() represent?

The Future represents the (potentially asynchronous) processing to obtain the next element. The Option represents the availability of the next element: Some(x) if another element is available, None if the enumeration is completed.