Firebase functions - retrieving files from Google cloud bucket results in RangeError: Maximum call stack size exceeded

Question

I am attempting to retrieve Firestore backup files from the Google cloud bucket in which they are stored:

export const retrieveFirestoreBackup = functions.https.onCall(
  async (data: RetrieveFirestoreBackupPayload, context) => {
      try {
        return await storage.bucket(bucket).getFiles();
      } catch (error) {
        console.log("Error getting backup files.", error);
        throw new functionsInit.https.HttpsError("unknown", error);
      }
  }
);

When I use my Firebase functions emulator, everything works as intended. But running the same code in the deployed Firebase functions, results in the following error:

Unhandled error RangeError: Maximum call stack size exceeded
    at isArrayLike (/srv/node_modules/lodash/lodash.js:11333:31)
    at keys (/srv/node_modules/lodash/lodash.js:13307:14)
    at /srv/node_modules/lodash/lodash.js:4900:21
    at baseForOwn (/srv/node_modules/lodash/lodash.js:2990:24)
    at Function.mapValues (/srv/node_modules/lodash/lodash.js:13400:7)
    at encode (/srv/node_modules/firebase-functions/lib/providers/https.js:179:18)
    at /srv/node_modules/lodash/lodash.js:13401:38
    at /srv/node_modules/lodash/lodash.js:4905:15
    at baseForOwn (/srv/node_modules/lodash/lodash.js:2990:24)
    at Function.mapValues (/srv/node_modules/lodash/lodash.js:13400:7)

How do I send the files to the client successfully?

Answer 1

Just posting my revised code, based on Doug Stevenson's suggestion, for those who need it:

export const retrieveFirestoreBackup = functions.https.onCall(
  async (data: RetrieveFirestoreBackupPayload, context) => {
    try {
        const [files] = await storage.bucket(bucket).getFiles();
        const metadataArray = await Promise.all(
          files.map(file => {
            return file.getMetadata().then(result => result[0]);
          })
        );
        return metadataArray;
      } catch (error) {
        console.log("Error getting backup files.", error);
        throw new functionsInit.https.HttpsError("unknown", error);
      }
  }
);

Answer 2

Your code is attempting to serialize the GetFilesResponse object returned by getFiles() as JSON to send back to the caller. The API documentation says it's an array of File objects.

The error message suggests that File objects are too complicated to send using the default JSON serialization. Note from the API documentation that they each contain a Bucket object, which itself probably contains lots of other implementation details. Take a good look at the inner details of a single File object if you want to get a sense of its complexity. Perhaps all this fits into memory on your machine, but not on the limited memory provided by the Cloud Functions server instance.

What you should probably do instead is just iterate the array of File objects from getFiles and build a new array of only the data about the file that you want to send back to the client. If all you need is the path of the file, that reduces the response down to a simple array of strings, which should OK for buckets with a limited number of files. Bear in mind the maximum size of a Cloud Functions response can only be 10MB, so for very large buckets, you could have a problem.