Cloud Function scheduler with memory allocation management

Question

I'm trying to create a scheduled function on Firebase Cloud Functions with third-party APIs. As the size of the data collected through the third-party API and passed to this scheduled function is huge, it returns Function invocation was interrupted. Error: memory limit exceeded. Function invocation was interrupted. Error: memory limit exceeded.

I have written this index.js (below) with help , but still looking for the way how it should handle the output data of large size inside the scheduler function.

index.js

const firebaseAdmin = require("firebase-admin");
const firebaseFunctions = require("firebase-functions");
firebaseAdmin.initializeApp();
const fireStore = firebaseAdmin.firestore();
const express = require("express");
const axios = require("axios");
const cors = require("cors");
const serviceToken = "SERVICE-TOKEN";
const serviceBaseUrl = "https://api.service.com/";

const app = express();
app.use(cors());

const getAllExamples = async () => {
    var url = `${serviceBaseUrl}/examples?token=${serviceToken}`;
    var config = {
        method: "get",
        url: url,
        headers: {}
    };
    return axios(config).then((res) => {
        console.log("Data saved!");
        return res.data;
    }).catch((err) => {
        console.log("Data not saved: ", err);
        return err;
    });
}

const setExample = async (documentId, dataObject) => {
    return fireStore.collection("examples").doc(documentId).set(dataObject).then(() => {
        console.log("Document written!");
    }).catch((err) => {
        console.log("Document not written: ", err);
    });
}

module.exports.updateExamplesRoutinely = firebaseFunctions.pubsub.schedule("0 0 * * *").timeZone("America/Los_Angeles").onRun(async (context) => {
    const examples = await getAllExamples(); // This returns an object of large size, containing 10,000+ arrays 
    const promises = [];
    for(var i = 0; i < examples.length; i++) {
        var example = examples[i];
        var exampleId = example["id"];
        if(exampleId && example) promises.push(setExample(exampleId, example));
    }
    return Promise.all(promises);
});

Firebase's official document simply tells how to set timeout and memory allocation manually as below, and I'm looking for the way how we should incorporate it with the above scheduler function.

exports.convertLargeFile = functions
    .runWith({
      // Ensure the function has enough memory and time
      // to process large files
      timeoutSeconds: 300,
      memory: "1GB",
    })
    .storage.object()
    .onFinalize((object) => {
      // Do some complicated things that take a lot of memory and time
    });

Answer 1

Firebase's official document simply tells how to set timeout and memory allocation manually as below, and I'm looking for the way how we should incorporate it with the above scheduler function .

You should do as follows:

module.exports.updateExamplesRoutinely = firebaseFunctions
    .runWith({
      timeoutSeconds: 540,
      memory: "8GB",
    })
   .pubsub
   .schedule("0 0 * * *")
   .timeZone("America/Los_Angeles")
   .onRun(async (context) => {...)

---

However you may still encounter the same error if you treat a huge number of "examples" in your CF. As you mentioned in the comment to the other answer it is advisable to cut it into chunks.

How to do that? It's highly depending on your specific case (ex: do you processes 10,000+ examples at each run? Or it is only going to happen once, in order to "digest" a backlog?).

You could treat only a couple of thousands of docs in the scheduled function and schedule it to run every xx seconds. Or you could distribute the work among several instances of the CF by using PubSub triggered versions of your Cloud Function.