ScriptDB: Custom ID for record for quick bulk updating?

Question

I have a 20,000 record ScriptDB of product information that I would like to have updated with current price/stock info on an ~hourly basis. As it stands I am having trouble designing something that doesn't hit against the execution limit headfirst.

Downloaded stockfile is customizeable, currently I have it as JSON, like:

[
{"sku":"MYSKU1","price":14.99,"qty":2},
{"sku":"MYSKU2","price":22.99,"qty":25},
{"sku":"MYSKU3","price":91.99,"qty":31}
]

Currently I do something like:

var prod_feed = UrlFetchApp.fetch("https://www.site.com/myJSONfile").getContentText();
var prod_data = JSON.parse(prod_feed);
var prod_qty = 0, i = 0;
var prod_code = "";
var prod_price = "";

var db = ScriptDb.getMyDb();
var result_array = [];
var result_current = {};
var time_obj = dbDateObject();

for(i = 0; i < prod_data.length; i++) {

    var result = db.query({sku: prod_data[i]["sku"]}).next(); // result will be null if no match

    if (!result) {
    // No match, create new object with this SKU for insertion
      result = {};
      result["sku"] = prod_data[i]["sku"];
    }

    result["qty"] = prod_data[i]["qty"];
    result["price_default"] = prod_data[i]["price"];
    result["last_product_update"] = time_obj;
    result_array[i] = result;

}

var results = db.saveBatch(result_array, false);

I can run through about 2,500 records this way, and save them, before hitting time limits.

Is there any way I can use the SKU directly as the record ID? Then it would be practically 3 lines of code...

var prod_feed = UrlFetchApp.fetch("https://www.site.com/myJSONfile").getContentText();
var prod_data = JSON.parse(prod_feed);
var results = db.saveBatch(prod_data, false);

If not, any obvious way to increase this efficiency?

Thanks!

Answer 1

If it's the 6 minute runtime limit that you're hitting then just (a) make your job idempotent and (b) run it again and again until your work is done.

To do (a) - you can "remember" what sku you're up to or have them as lineitems in a spreadsheet and add a "done" column.

To do (b), in your main loop, if you're over 5 mins and under 6 mins, you can quit and reschedule the job to run again in 10 mins (or less but you'll need locks then).

Answer 2

Is there any way I can use the SKU directly as the record ID?

Sadly, no. Every ScriptDb object has an id, assigned by the system when it is stored, which is an opaque, immutable string value. ( ScriptDb documentation .)

However, instead of an array of objects, you could store your products in a single object, with the SKU used as an object attribute. You'll still need to loop through the input, but you'll only have 2 calls to the ScriptDB service instead of thousands.

Try this:

  var prod_feed = UrlFetchApp.fetch("https://www.site.com/myJSONfile").getContentText();
  var prod_data = JSON.parse(prod_feed);

  var db = ScriptDb.getMyDb();
  var result = db.query({type: "products"});
  var products = {};
  if (result.hasNext()) {
    // Projects database object already exists
    products = result.next();
  }
  else {
    // Not found, so create the Projects database object.
    var product_data = new Object();
    products = {type: "products", data: product_data};
  }

  var time_obj = dbDateObject();

  for (var i = 0; i < prod_data.length; i++) {

    if (!(prod_data[i]["sku"] in products.data)) {
      // No match, create new object with this SKU for insertion
      products.data[prod_data[i]["sku"]] = {};
    }

    // Update info for this item
    var item = products.data[prod_data[i]["sku"]];
    item.qty = prod_data[i].qty;
    item.price_default = prod_data[i].price;
    item.last_product_update = time_obj;
  }

  var results = db.save(products);

Here's a screenshot from the debugger, showing what the "products" object looks like when retrieved from the ScriptDb. If you could massage your input data a bit, you might be able to get that one-pass write you're dreaming of!

Note: I've used price and price_default as you did... but I suspect that's a typo.