Unable to Update Cosmos DB collection by Azure Stream Analytics Job when TTL is set as ON for the collection

Question

I have a stream analytics job which will dump the data from a Blob to Cosmos DB.

While I am running the job I am getting the below error and the Cosmos DB collection is not getting updated.

Error:

[10:20:57 AM] Source 'signalData' had 1 occurrences of kind 'OutputDataConversionError.RequiredColumnMissing' between processing times '2019-06-27T04:50:57.4433960Z' and '2019-06-27T04:50:57.4433960Z'.

Below is the SQL in Stream analytic job.

SELECT
    *
INTO
    [signalData]
FROM
    [signalDataBlob]

This Stream analytics function is working fine and updating cosmos collection if I am keeping TTL as OF. Issue is only when TTL is ON. The compatibility level selected for Azure Stream analytics Job is v1.1.

This is not a duplicate question since there are so many questions related to this error. But in my case it is happening only when the value of TTL changes from OFF to ON.

Below is the data I am trying to update into cosmos DB.

[
{
        "id": "Z5002XXXXX",
        "type": "SIGNAL",
        "time": 1557302250,
        "abc": 35.0,
        "def": 136,
        "ijk": 161,
        "xyz": 88,
        "id": "432a0719-e1c8-e53a",
        "_rid": "3rEvAJlPHSA==",
        "_self": "dbs/3rEvAA==/colls/3rEvAJlPHSA=/docs/3rEvAJlPHSA==/",
        "_etag": "\"2600ce55-0000-2300-0000-5cf1129\"",
        "_attachments": "attachments/",
        "_ts": 1559302815
    }
]

Answer 1

1. Are you setting TTL on the container level ?
2. What value is it? If value is too low, the items could expire before you detect.
3. By "updated" you mean there are no events landing in CosmosDB ?
4. Are there any output events in Stream Analytics job (portal metric)?
5. Select * and RequiredColumnMissing are difficult to diagnose as you may have different columns per event or the case might be different. Can you please change it to exact column names for troubleshooting?