FIWARE - Orion Context Broker offset parameter behaviour

Question

I am facing the following problem regarding FIWARE Orion Context Broker, I hope someone has an idea about it.

I store records in FIWARE Orion CB, version 1.2.0, running in a Docker instance and in one type I receive the following response after posting a GET call to http://mywebsite.eu:1016/v2/types/MyTYPE

{
    "attrs": {
        "cid": {
            "types": [
                "String"
            ]
        },
        "datetime": {
            "types": [
                "String"
            ]
        },
        "humidity": {
            "types": [
                "Float"
            ]
        },
        "luminosity": {
            "types": [
                "Integer"
            ]
        },
        "temperature": {
            "types": [
                "Float"
            ]
        }
    },
    "count": 55811
}

As you can see the “count” is 55811. But, when I send another GET call to http://mywebsite.eu:1026/v2/entities?type=MyTYPE&offset=54908&limit=1 , I receive the following:

[
    {
        "id": ".*",
        "type": "MyTYPE"
    }
]

From that number of offset (54908) and up, the response is the same. I've checked space in my server and there is plenty of it, so it is not a matter of disk space. I've checked that data is going to Orion CB the same way as before from my Raspberries. So, my question is if there is an upper limit for records per type that Orion can store and when this limit is reached I should change type. Maybe there is something that I am missing and any advice you can give me will help me a lot.

Answer 1

The offset parameter means how many elements have to be skipped before returning results. Thus, offset=0 (the defaulf value if offset parameter is ommited) means to start in the first element, offset=1 means to start in the second element, an so.

Let's consider the ASN type, which has 54879 entities (as GET /v2/types/ASN shown). Using offset=0 we get the first entity (which happens to be the one with id 1470515373636_1 ):

GET /v2/entities?type=ASN&limit=1

[
  {
    "id": "1470515373636_1"
    ...
  }
]

Let's use now offset 54878 (equal to the total number of ASN entities minus one). That is, skipping the first 54878 entities only the last one remains (which id happens to be 1480344938807_1 ):

GET /v2/entities?type=ASN&offset=54878&limit=1

[
  {
    "id": "1480344938807_1"
    ...
  }
]

Note that if we use offset equal to the total number of entities (or any greater number), ie 54879, that means that all the entities have been skipped. In other words, we are going beyond the limit. Thus is is normal that in that case Orion return an empty list of entities:

GET /v2/entities?type=ASN&offset=54879&limit=1

[]

Another "consistency" check: if you invert the order (by default, entities are ordered by increasing creation date, ie from older to newer, but you can change that using the orderBy parameter) you can check that the former first element is now the last and viceversa. In particular, orderBy=!dateCreated orders results by decreasing creation date (ie from newer to older).

Thus, now searching for the first element return the former last (ie 1480344938807_1 )

GET /v2/entities?type=ASN&limit=1&orderBy=!dateCreated

[
  {
    "id": "1480344938807_1"
    ...
  }
]

and searching for the last element return the former last (ie 1470515373636_1 )

GET /v2/entities?type=ASN&offset=54878&limit=1&orderBy=!dateCreated

[
  {
    "id": "1470515373636_1"
    ...
  }
]

Let's do another test. Let's add a new ASN entity, just for fun:

POST /v2/entities?options=keyValues

{
  "id": "TestingEntity",
  "type": "ASN",
  "A": 42
}

so, now the total number of entities in the ASN type is 54880. The new entity is the last one to be created, so it will be put at the end with default ordering (ie increasing creation date). You can check it with the following query:

GET /v2/entities?type=ASN&offset=54879&limit=1

[
  {
    "id": "TestingEntity"
    ...
  }
]

Note that that query returned [] before adding the new entity (see above).

In sum, Orion Context Broker behaviour seems to be as expected with regard to offset parameter. You say that the problem was Orion won't return proper results after a specific offset number, but note that empty results is a proper result if your offset has passed beyond the total number of elements.

Answer 2

It has been observed that with large offset values you may get the following effect:

• The response you get is the one that the question post mentions, eg:

   [ { "id": ".*", "type": "ASN" } ] 

• An error message like this one appears in Orion logs:

   Raising alarm DatabaseError: nextSafe(): { $err: "Executor error: OperationFailed Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.", code: 17144 } 

The solution is creating an index in the DB field used for sorting. In the case of default ordering (based on creation date) the index can be created at Mongo shell as follows (assuming that orion is the name of the DB your are using):

$ mongo
> use orion
> db.entities.createIndex({creDate: 1})

In fact, it uses to be a good idea to create the indexes described in this section in the documentation .

EDIT : the response is not the proper one: Orion should response using a 500 Internal Server Error with a descriptive message about the problem. This fix is ready to land in master branch and will be included in Orion 1.7.0 (to be released by the early 2017).