Mongoose findOneAndUpdate: create and then update nested array

Question

I have a program where I'm requesting weather data from a server, processing the data, and then saving it to an mlab account using mongoose. I'm gathering 10 years of data, but the API that I'm requesting the data from only allows about a year at a time to be requested.

I'm using findOndAndUpdate to create/update the document for each weather station, but am having trouble updating the arrays within the data object. (Probably not the best way to describe it...)

For example, here's the model:

    const stnDataSchema = new Schema(
       { 
        station: { type: String, default: null }, 
        elevation: { type: String, default: null },
        timeZone: { type: String, default: null },
        dates: {},
        data: {}
       }, 
       { collection: 'stndata' },
       { runSettersOnQuery: true }
     )

where the dates object looks like this:

    dates: ["2007-01-01",
    "2007-01-02",
    "2007-01-03",
    "2007-01-04",
    "2007-01-05",
    "2007-01-06",
    "2007-01-07",
    "2007-01-08",
    "2007-01-09"]

and the data object like this:

    "data": [
       {
        "maxT": [
            0,
            null,
            4.4,
            0,
            -2.7,
            etc.....

what I want to have happen is when I run findOneAndUpdate I want to find the document based on the station, and then append new maxT values and dates to the respective arrays. I have it working for the date array, but am running into trouble with the data array as the elements I'm updated are nested. I tried this:

    const update = {
    $set: {'station': station, 'elevation': elevation, 'timeZone': timeZone},
    $push: {'dates': datesTest, 'data.0.maxT': testMaxT}};
    StnData.findOneAndUpdate( query, update, {upsert: true} ,
    function(err, doc) {
    if (err) {
     console.log("error in updateStation", err)
     throw new Error('error in updateStation')
   }
   else {
     console.log('saved')

but got an output into mlab like this:

    "data": {
      "0": {
          "maxT": [
            "a",
            "b",

the issue is that I get a "0" instead of an array of one element. I tried 'data[0].maxT' but nothing happens when I do that.

The issue is that the first time I run the data for a station, I want to create a new document with data object of the format in my third code block, and then on subsequent runs, once that document already exists, update the maxT array with new values. Any ideas?

Answer 1

You are getting this output:

 "data": {
    "0": {
      "maxT": [
        "a",
        "b",

because you are upserting the document. Upserting gets a bit complicated when dealing with arrays of documents.

When updating an array, MongoDB knows that data.0 refers to the first element in the array. However, when inserting, MongoDB can't tell if it's meant to be an array or an object. So it assumes it's an object. So rather than inserting ["val"] , it inserts {"0": "val"} .

---

Simplest Solution

Don't use an upsert. Insert a document for each new weather station then use findOndAndUpdate to push values into the arrays in the documents. As long as you insert the arrays correctly the first time, you will be able to push to them without them turning into objects.

---

Alternative Simple Solution if data just Contains one Object

From your question, it looks like you only have one object in data . If that is the case, you could just make the maxT array top-level, instead of being a property of a single document in an array. Then it would act just like dates .

---

More Complicated MongoDB 3.6 Solution

If you truly cannot do without upserts, MongoDB 3.6 introduced the filtered positional operator $[<identifier>] . You can use this operator to update specific elements in an array which match a query. Unlike the simple positional operator $ , the new $[<identifier>] operator can be used to upsert as long as an exact match is used.

You can read more about this operator here: https://docs.mongodb.com/manual/reference/operator/update/positional-filtered/

So your data objects will need to have a field which can be matched exactly on (say name ). An example query would look something like this:

let query = {
  _id: 'idOfDocument', 
  data: [{name: 'subobjectName'}] // Need this for an exact match
}

let update = {$push: {'data.$[el].maxT': testMaxT}}

let options = {upsert: true, arrayFilters: [{'el.name': 'subobjectName'}]}

StnData.findOneAndUpdate(query, update, options, callbackFn)

As you can see this adds much more complexity. It would be much easier to forget about trying to do upserts. Just do one insert then update.

Moreover mLab currently does not support MongoDB 3.6. So this method won't be viable when using mLab until 3.6 is supported.