How do I make node.js MongoDB follow object references?

Question

This question is invalid.

See my comment .

---

When you have built a complex object with multidimensional arrays in javascript, MongoDB does not store any array that is technically a reference. MongoDB does not follow the reference and store the data in the document.

Is there a way to force this?

---

For example, an object contains an array called products . Every product has an array called suppliers . And every supplier has an array called shipments .

This object is dynamically created, and hence the sub-arrays are references to the actual array in the parent object (by javascript design).

Only the first array is stored in MongoDB, the sub-domains are ignored.

To fix this, I have to clone the entire object. Eg: jQuery.extend(true, {}, object); . But this can take up to a second, which is a long time, and I feel like it itsn't necessary.

How can I feed this object with it's references into MongoDB? Is there a faster way to clone the object and remove the references? Is there some other solution?

I am looking for specifically this scenario. Not for ideas on how to restructure my data.

---

Internally, javascript creates a reference to every array or object that is created. If you assign that array to an object value, the memory reference gets assigned; not the actual contents.

Main object

{
    a : 33,
    lot : 52,
    of : true,
    data : false,
    products : [Array] (id=115)
}

products

products : [
    {
        more : true,
        data : false,
        suppliers : [Array] (id=612)
    },
    {},
    {},
    //...
]

suppliers

suppliers : [
    {
        even : null,
        more : 52,
        data : false,
        shipments : [Array] (id=854)
    },
    {},
    {},
    //...
]

etc.

The first array is inserted into MongoDB; the rest is missing.

After flattening (jQuery.extend), like this:

{
    a : 33,
    lot : 52,
    of : true,
    data : false,
    products : [
        {
            more : true,
            data : false,
            suppliers : [
                {
                    even : null,
                    more : 52,
                    data : false,
                    shipments : [
                        {
                            some : 'foo',
                            more : 'bar',
                            data : false
                        },
                        {},
                        {},
                        //...
                    ]
                },
                {},
                {},
                //...
            ]
        },
        {},
        {},
        //...
    ],
}

The document gets inserted into MongoDB without problem.

Answer 1

This is the issue of shallow copy vs. deep copy.

While you could iterate through every key in the object at runtime and check for arrays (or arrays of arrays, etc.), that's probably not the most performant. If you already know your object structure, then you should be able to build a deep copy.

function Foo() {
     this.prop = "someKey";
     this.arr2d = [[0],[1],[2]];
}

Foo.prototype.getDeepCopy = function() {
    var self = this;
    return {
        prop: self.prop,
        arr2d: function() {
            var arr = [];
            for ( var i = 0; i < self.arr2d.length; i++ ) {
                var _a = [];
                for ( var j = 0; j < self.arr2d[i].length; j++ ) {
                    _a[j] = self.arr2d[i][j];
                }
                arr[i] = _a;
            }
            return arr;
        }()
    };
}

You could also do this with a function that takes object references or literals instead of using a prototype.

function deepCopy(obj) {
    return {
        prop:obj.prop,
        arr2d:function() {...}()
    };
}

Deep copies will still need to iterate over the object structure, but it's going to be faster than doing

var copy = JSON.parse(JSON.stringify(obj));

Edit: In case it's not clear, you will have to build a datum without references -- that means either rebuilding it via a deep copy, or constructing it literally in the first place.

Answer 2

This question is invalid. MongoDB does follow the references. I couldn't make a script that reproduces this behavior.

I figured it out in my main app where I witnessed this behavior. A few promises down the pipeline, the object-to-be-stored get's altered. Removing some of the arrays.

Since I wasn't waiting for MongoDB to call back (because a failure to write doesn't mean the rest of the process shouldn't continue), apparently the object referenced was altered before MongoDB started to write this to the database.

This was totally outside of my frame of reference when debugging this. I thought that each promised step would be executed on a next tick of the event loop . I assumed the node MongoDB module would have passed the data on to the adapter before it changed a few ticks later.

I need to enable write concern and continue with the object only after the callback is launched. This takes a bit longer, but I won't have to clone the object so that will save time.