Enable MongoDB sharding for FIWARE Orion ContextBroker

Question

I am trying to setup MongoDB sharding for the Orion ContextBroker, and IotAgent-Json, I am able to distribute the device provisioning and service data in MongoDb between shards, but whenever i sends the actual device data it gives following error:

{ "name": "ENTITY_GENERIC_ERROR", "message": "Error accesing entity data for device: DeviceEntity1 of type: DeviceEntity1"}

I am using following command to shard the collection:
sh.shardCollection("orion-fiwareiot.entities",{"_id.servicePath": 1, "_id.id": 1, "_id.type": 1})

Intercept of ContextBroker log:

time=Wednesday 25 Sep 06:54:22 2019.729Z | lvl=ERROR | corr=4e22de66-df61-11e9-aa53-3464a92017cd | trans=1569393389-597-00000000007 | from=127.0.0.1 | srv=fiwareiot | subsrv=/raw | comp=Orion | op=AlarmManager.cpp[211]:dbError | msg=Raising alarm DatabaseError: collection: orion-fiwareiot.entities - update(): <{ _id.id: "DeviceEntity1", _id.type: "DeviceEntity", _id.servicePath: { $in: [ /^/raw$/ ] } },{ $set: { attrs.deviceName: { value: "FR", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.locationType: { value: "O", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.locationName: { value: "Mumbai", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.deviceType: { value: "FR", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.deviceMa cAddress: { value: "84:24:8D:F8:B2:B2", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.TimeInstant: { value: 1569385065.0, type: "ISO8601", mdNames: [], creDate: 1569384878, modDate: 1569385065 }, modDate: 1569385065, lastCorrelator: "6d0aa4fa-df4b-11e9-8293-3464a92017cd" }, $unset: { location: 1, expDate: 1 } }> - exception: OperationException: { index: 0, code: 61, errmsg: "update { q: { _id.id: "DeviceEntity1", _id.type: "DeviceEntity", _id.servicePath: { $in: [ /^/raw$/ ] } }, u: { $set: { attrs.deviceName: { v...", op: { q: { _id.id: "DeviceEntity1", _id.type: "DeviceEntity", _id.servicePath: { $in: [ /^/raw$/ ] } }, u: { $set: { attrs.deviceName: { value: "FR", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.locationType: { value: "O", t ype: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.locationName: { value: "Mumbai", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.deviceType: { value: "FR", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.deviceMacAddress: { value: "84:24:8D:F8:B2:B2", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878, modDate: 1569385065 }, attrs.TimeInstant: { value: 1569385065.0, type: "ISO8601", mdNames: [], creDate: 1569384878, modDate: 1569385065 }, modDate: 1569385065, lastCorrelator: "6d0aa4fa-df4b-11e9-8293-3464a92017cd" }, $unset: { location: 1, expDate: 1 } }, multi: false, upsert: false } }

I also tried to manually update the document, the log is:

update { q: { _id.id: "DeviceEntity1", _id.type: "DeviceEntity", _id.servicePath: { $in: [ /^/raw$/ ] } }, u: { $set: { attrs.deviceName: { value: "FR", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878.0, modDate: 1569385065.0 }, attrs.locationType: { value: "O", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878.0, modDate: 1569385065.0 }, attrs.locationName: { value: "Mumbai", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878.0, modDate: 1569385065.0 },attrs.deviceType: { value: "FR", type: "string", md: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878.0, modDate: 1569385065.0 }, attrs.deviceMacId: { value: "84:24:8D:F8:B2:B2", type: "string", m d: { TimeInstant: { type: "ISO8601", value: "2019-09-25T04:17:45.645Z" } }, mdNames: [ "TimeInstant" ], creDate: 1569384878.0, modDate: 1569385065.0 }, attrs.TimeInstant: { value: 1569385065.0, type: "ISO8601", mdNames: [], creDate: 1569384878.0, modDate: 1569385065.0 }, modDate: 1569385065.0, lastCorrelator: "6d0aa4fa-df4b-11e9-8293-3464a92017cd" }, $unset: { location: 1.0, expDate: 1.0 } }, multi: false, upsert: false } does not contain _id or shard key for pattern { _id.servicePath: 1.0, _id.id: 1.0, _id.type: 1.0 }

What MongoDB shard key strategy should i follow, for following composite document id, there may be multiple document for _id.id and _id.servicePath . I want to distribute data with "_id.id": "DeviceEntity1" and "_id.id": "DeviceEntity2" in two MongoDB shards.

    "_id" : {
        "id" : "DeviceEntity1",
        "type" : "DeviceEntity",
        "servicePath" : "/raw"
    },... 

Thank You.

Answer 1

I'm not fully sure (a long time has passed since my last MongoDB sharding set up:) but I'd suggest trying first with a simpler sharding key. For instance:

db.shardCollection("orion-fiwareiot.entities", {"_id.id": 1})

and see how it goes.

With regards to:

I want to distribute data with "_id.id": "DeviceEntity1" and "_id.id": "DeviceEntity2" in two MongoDB shards.

Maybe I'm not remembering correctly or it has changed in a recent MongoDB version but, as far as I know, you cannot choose where you particular documents go. You set a sharding key and MongoDB sharding system does all the magic to distribute your documents among shards, based on the key range. But you cannot ensure that a particular document ends in a particular shard in a given time.