Error 500 in Orion CB when creating a new FiwareService

Question

I'm using Fiware Orion CB version 2.1.0 on top of a MongoDB 4.0.5. Everything seems to work fine once I've started it, but, after a while (a copule of days) ingesting data with no problems, now, everytime I try to create a new entity in a new fiwareservice I get an 500 error from OrionCB (if I try to create the same entity in an exisiting FS, it works), and I have to reboot both, Mongo and Orion containers to get this new FS creation works.

The request is:

curl --location --request POST 'context.infinitech.easier-ai.eu/v2/entities/' \
--header 'fiware-service: itscenterctag' \
--header 'fiware-servicepath: /vehicle' \
--header 'Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--header 'Content-Type: application/json' \
--data-raw ' {
        "id": "TEST",
        "type": "test",
        "dateObserved": {
            "type": "DateTime",
            "value": "2020-10-02T14:00:00.00Z",
            "metadata": {}
        },
        "test": {
            "type": "Number",
            "value": 6.1,
            "metadata": {
                "timestamp": {
                    "type": "DateTime",
                    "value": "2020-10-02T14:00:00.00Z"
                }
            }
        },
        "location": {
            "type": "geo:json",
            "value": {
                "type": "Point",
                "coordinates": [
                    -7.733611,
                    42.079166
                ]
            },
            "metadata": {}
        },
        "name": {
            "type": "Text",
            "value": "test",
            "metadata": {}
        }
    }'

Orion Response:

{
    "error": "InternalError",
    "description": "Database Error (collection: orion-itscenterctag.entities - insert(): { _id: { id: "TEST", type: "test", servicePath: "/vehicle" }, attrNames: [ "dateObserved", "test", "location", "name" ], attrs: { dateObserved: { type: "DateTime", creDate: 1603208149, modDate: 1603208149, value: 1601647200.0, mdNames: [] }, test: { type: "Number", creDate: 1603208149, modDate: 1603208149, value: 6.1, md: { timestamp: { type: "DateTime", value: 1601647200.0 } }, mdNames: [ "timestamp" ] }, location: { type: "geo:json", creDate: 1603208149, modDate: 1603208149, value: { type: "Point", coordinates: [ -7.733611, 42.079166 ] }, mdNames: [] }, name: { type: "Text", creDate: 1603208149, modDate: 1603208149, value: "test", mdNames: [] } }, creDate: 1603208149, modDate: 1603208149, location: { attrName: "location", coords: { type: "Point", coordinates: [ -7.733611, 42.079166 ] } }, lastCorrelator: "ec0dba3a-12e9-11eb-a65d-52e1fcc227da" } - exception: OperationException: { index: 0, code: 8, errmsg: "1: Operation not permitted", op: { _id: { id: "TEST", type: "test", servicePath: "/vehicle" }, attrNames: [ "dateObserved", "test", "location", "name" ], attrs: { dateObserved: { type: "DateTime", creDate: 1603208149, modDate: 1603208149, value: 1601647200.0, mdNames: [] }, test: { type: "Number", creDate: 1603208149, modDate: 1603208149, value: 6.1, md: { timestamp: { type: "DateTime", value: 1601647200.0 } }, mdNames: [ "timestamp" ] }, location: { type: "geo:json", creDate: 1603208149, modDate: 1603208149, value: { type: "Point", coordinates: [ -7.733611, 42.079166 ] }, mdNames: [] }, name: { type: "Text", creDate: 1603208149, modDate: 1603208149, value: "test", mdNames: [] } }, creDate: 1603208149, modDate: 1603208149, location: { attrName: "location", coords: { type: "Point", coordinates: [ -7.733611, 42.079166 ] } }, lastCorrelator: "ec0dba3a-12e9-11eb-a65d-52e1fcc227da" } })"
}

From MongoDB log I get:

2020-10-20T15:35:49.592+0000 E STORAGE  [conn1] WiredTiger error (1) [1603208149:592173][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-18--6388608708582174430.wt: handle-open: open: Operation not permitted Raw: [1603208149:592173][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-18--6388608708582174430.wt: handle-open: open: Operation not permitted
2020-10-20T15:35:49.592+0000 E -        [conn1] Assertion: UnknownError: 1: Operation not permitted src/mongo/db/catalog/database_impl.cpp 831
2020-10-20T15:35:49.597+0000 I STORAGE  [conn1] createCollection: orion-itscenterctag.entities with generated UUID: a9c4273f-cddd-42bf-af6b-cbd96cbe034d
2020-10-20T15:35:49.598+0000 E STORAGE  [conn1] WiredTiger error (1) [1603208149:598543][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-19--6388608708582174430.wt: handle-open: open: Operation not permitted Raw: [1603208149:598543][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-19--6388608708582174430.wt: handle-open: open: Operation not permitted
2020-10-20T15:35:49.598+0000 E -        [conn1] Assertion: UnknownError: 1: Operation not permitted src/mongo/db/catalog/database_impl.cpp 831
2020-10-20T15:35:49.598+0000 I STORAGE  [conn1] createCollection: orion-itscenterctag.entities with generated UUID: 49b14431-f217-48e7-93ed-d2f2f8e7a4fc
2020-10-20T15:35:49.600+0000 E STORAGE  [conn1] WiredTiger error (1) [1603208149:600253][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-20--6388608708582174430.wt: handle-open: open: Operation not permitted Raw: [1603208149:600253][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-20--6388608708582174430.wt: handle-open: open: Operation not permitted
2020-10-20T15:35:49.600+0000 E -        [conn1] Assertion: UnknownError: 1: Operation not permitted src/mongo/db/catalog/database_impl.cpp 831
2020-10-20T15:35:49.600+0000 I STORAGE  [conn1] createCollection: orion-itscenterctag.entities with generated UUID: 600ec5a0-6841-4948-93c8-4fd062223479
2020-10-20T15:35:49.601+0000 E STORAGE  [conn1] WiredTiger error (1) [1603208149:601495][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-21--6388608708582174430.wt: handle-open: open: Operation not permitted Raw: [1603208149:601495][1:0x7ff40cab4700], WT_SESSION.create: __posix_open_file, 715: /data/db/collection-21--6388608708582174430.wt: handle-open: open: Operation not permitted
2020-10-20T15:35:49.601+0000 E -        [conn1] Assertion: UnknownError: 1: Operation not permitted src/mongo/db/catalog/database_impl.cpp 831

UPDATE:

This ONLY happens when trying to create a new FiwareService (by creating a new entity in a new FS). Managing any exisiting entity within already created FS works fine.

UPDATE II: (According comments)

> use orion-foo
switched to db orion-foo
> db.entities.insert({x:1})
WriteResult({
        "nInserted" : 0,
        "writeError" : {
                "code" : 8,
                "errmsg" : "1: Operation not permitted"
        }
})

Answer 1

Taking into account the inforamtion in UPDATEII, it seems the problem is in the MongoDB instance itself, not in Orion. I mean, the fail got at mongo shell shows that the database itself is not allowing to create new databases (no matter who is creating them, either Orion, you at the mongo shell or a third application connected to MongoDB).

Thus, I'd suggest to review your database setup. What is a bit suprissing is that according to the report it works at the beginning but it start to fail after a while. Maybe some out-of-band operations (ie an administration process) is changing permissions on the move?