ReactiveMongo database dump with Play Framework 2.5

Question

I'm trying to dump my mongo database into a json object but because my queries to the database are asynchrounous I'm having problems.

Each collection in my database contains user data and each collection name is a user name.

So, when I want to get all my users data I recover all the collection names and then loop over them to recover each collection one by one.

def databaseDump(prom : Promise[JsObject]) = {
    for{
      dbUsers <- getUsers
    } yield dbUsers

var rebuiltJson = Json.obj()
var array = JsArray()
res.map{ users =>
  users.map{ userNames =>
    if(userNames.size == 0){
      prom failure new Throwable("Empty database")
    }
    var counter = 0
    userNames.foreach { username =>
      getUserTables(username).map { tables =>
         /* Add data to array*/
           ...
        counter += 1
        if(counter == userNames.size){
          /*Add data to new json*/
             ...
          prom success rebuiltJson
        }

      } 
    }
  }
}

This kinda works, but sometimes the promise is succesfully triggered even though all the data has not yet been recoverd. This is due to that fact that my counter variable is not a reliable solution.

Is there a way to loop over all the users, query the database and wait for all the data to be recovered before succesfully triggering the promise? I tried to use for comprehension but didn't find a way to do it. Is there a way to dump a whole mongo DB into one Json : { username : data, username : data ..} ?

Answer 1

The users/tables terminology was getting me confused, so I wrote a new function that dumps a database into a single JsObject .

// helper function to find all documents inside a collection c
// and return them as a single JsArray
def getDocs(c: JSONCollection)(implicit ec: ExecutionContext) = c.find(Json.obj()).cursor[JsObject]().jsArray()

def dumpToJsObject(db: DefaultDB)(implicit ec: ExecutionContext): Future[JsObject] = {
  // get a list of all collections in the db
  val collectionNames = db.collectionNames
  val collections = collectionNames.map(_.map(db.collection[JSONCollection](_)))

  // each entry is a tuple collectionName -> content (as JsArray)
  val namesToDocs = collections.flatMap {
    colls => Future.sequence(colls.map(c => getDocs(c).map(c.name -> _)))
  }

  // convert to a single JsObject
  namesToDocs.map(JsObject(_))
}

I haven't tested it yet (I will do so later), but this function should at least give you the general idea. You get the list of all collections inside the database. For each collection, you perform a query to get all documents inside that collection. The list of documents is converted into a JsArray , and finally all collections are composed to a single JsObject with the collection names as keys.

Answer 2

If the goal is to write the data to an output stream (local/file or network), with side effects.

import scala.concurrent.{ ExecutionContext, Future }
import reactivemongo.bson.BSONDocument
import reactivemongo.api.{ Cursor, MongoDriver, MongoConnection }

val mongoUri = "mongodb://localhost:27017/my_db"

val driver = new MongoDriver
val maxDocs = Int.MaxValue // max per collection

// Requires to have an ExecutionContext in the scope
// (e.g. `import scala.concurrent.ExecutionContext.Implicits.global`)
def dump()(implicit ec: ExecutionContext): Future[Unit] = for {
  uri <- Future.fromTry(MongoConnection.parseURI(mongoUri))
  con = driver.connection(uri)
  dn <- Future(uri.db.get)
  db <- con.database(dn)
  cn <- db.collectionNames
  _  <- Future.sequence(cn.map { collName =>
    println(s"Collection: $collName")

    db.collection(collName).find(BSONDocument.empty). // findAll
      cursor[BSONDocument]().foldWhile({}, maxDocs) { (_, doc) =>
        // Replace println by appropriate side-effect
        Cursor.Cont(println(s"- ${BSONDocument pretty doc}"))
      }
  })
} yield ()

If using with the JSON serialization pack, just replace BSONDocument with JsObject (eg BSONDocument.empty ~> Json.obj() ).

If testing from the Scala REPL, after having paste the previous code, it can be executed as following.

dump().onComplete {
  case result =>
    println(s"Dump result: $result")
    //driver.close()
}