如何将Upsert json文件批量添加到Azure Cosmos DB（documentDB模块）？

Question

I am using python to update a lot of data files with new observations using the documentDB module. I have to upsert 100-200 json files per minute, and the upserting operation takes up way more time than the rest of the program. Right now I'm using the 'UpsertDocument' function from the DocumentClient in the module. Is there a faster/better way?

Answer 1

You can use stored procedure for the bulk upsert operation:

function bulkimport2(docObject) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();

// The count of imported docs, also used as current doc index.
var count = 0;

getContext().getResponse().setBody(docObject.items);
//return

// Validate input.
//if (!docObject.items || !docObject.items.length) getContext().getResponse().setBody(docObject);
docObject.items=JSON.stringify(docObject.items)
docObject.items = docObject.items.replace("\\\\r", "");
docObject.items = docObject.items.replace("\\\\n", "");
var docs = JSON.parse(docObject.items);
var docsLength = docObject.items.length;
if (docsLength == 0) {
    getContext().getResponse().setBody(0);
    return;
}

// Call the CRUD API to create a document.
tryCreate(docs[count], callback, collectionLink,count);

// Note that there are 2 exit conditions:
// 1) The createDocument request was not accepted.
//    In this case the callback will not be called, we just call setBody and we are done.
// 2) The callback was called docs.length times.
//    In this case all documents were created and we don't need to call tryCreate anymore. Just call setBody and we are done.
function tryCreate(doc, callback, collectionLink,count ) {
    doc=JSON.stringify(doc);
    if (typeof doc == "undefined") {
        getContext().getResponse().setBody(count);
        return ;
        } else {
        doc = doc.replace("\\r", "");
        doc = doc.replace("\\n", "");
        doc=JSON.parse(doc);
       }

    getContext().getResponse().setBody(doc);

    var isAccepted = collection.upsertDocument(collectionLink, doc, callback);

    // If the request was accepted, callback will be called.
    // Otherwise report current count back to the client, 
    // which will call the script again with remaining set of docs.
    // This condition will happen when this stored procedure has been running too long
    // and is about to get cancelled by the server. This will allow the calling client
    // to resume this batch from the point we got to before isAccepted was set to false
    if (!isAccepted) {
        getContext().getResponse().setBody(count);
     }
}

// This is called when collection.createDocument is done and the document has been persisted.
function callback(err, doc, options) {
    if (err) throw getContext().getResponse().setBody(err + doc);

    // One more document has been inserted, increment the count.
    count++;

    if (count >= docsLength) {
        // If we have created all documents, we are done. Just set the response.
        getContext().getResponse().setBody(count);
        return ;
    } else {
        // Create next document.
        tryCreate(docs[count], callback,  collectionLink,count);
    }
}

and then you can load in python and execute it. Please note that Stored procedure requires partitioning key.

Hope it helps.

Answer 2

One option would be to use the Cosmos DB Spark connector instead, and, optionally (and conveniently) run as a job in Azure Databricks. This will provide a significant amount of control over your throughput and make it easy to find the optimal balance between parallelism (which I think is the issue) and RU capacity on the Cosmos DB.

Here's a simple example of measurements taken loading 118K documents, and this is using a minimum spec Databricks cluster with just 1 worker.

Single Cosmos client in Python: 28 docs/sec @ 236 RUs (ie not pushing Cosmos at all)

Spark Cosmos DB Adapter, 66 docs/sec @ >400 RUs (was throttled due to 400 RU limit)

... after bumping up Cosmos DB to 10K RUs Spark Cosmos DB Adapter, 1317 docs/sec @ >2.9K RUs (don't think it ran long enough for accurate RUs) - still the same minimum spec cluster

You could also try Python multi-threading (I think it will help), and as CYMA said in the comment, you should be checking for throttling at Cosmos DB. My observation, though, is a single Cosmos client isn't going to get you even to the minimum 400 RUs.