How to import 300Mb JSON file from Firebase Storage to Database using 'onFinalize' in Cloud Functions?

Question

I'm trying to import five JSON files from firebase storage to database using cloud function fired by five 'onFinalize' events after they uploaded each after other. Each of them has it's own collection.

JSON files: usualy it's a 5 files (~13Mb with ~9000 records, ~19Mb with ~21000 records, ~23Mb with 57000 records, ~95Mb with 73000 records and ~435Mb with 208000 records) with total size of all files ~600Mb

My code working fine for very small files (I tested on sample 1Kb file), but when I'm trying to upload at least 13Mb in firebase console I'm getting next error:

1. Unhandled rejection
2. Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded at Object.exports.createStatusError (/srv/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:1204:28) at InterceptingListener._callNext (/srv/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/srv/node_modules/grpc/src/client_interceptors.js:845:24)

and it adds to database only one first object. It looks like function becomes recursive, but I don't understand why :(

const functions = require('firebase-functions')
const { Storage } = require('@google-cloud/storage')
const storage = new Storage()
const admin = require('firebase-admin')
admin.initializeApp(functions.config().firebase)
const db = admin.firestore()
const path = require('path')

exports.logNewJSONFiles = functions
    .runWith({ timeoutSeconds: 120, memory: '2GB' })
    .storage.object()
    .onFinalize((file) => {
        return new Promise(function(resolve, reject){
            storage
                .bucket(file.bucket)
                .file(file.name)
                .download()
                .then(function(data){
                    if (data) return JSON.parse(data)
                })
                .then(function(data){
                    if (data) {
                        const destination = db.collection(path.basename(file.name))
                        for (var obj of data) destination.doc().set(obj)
                    }
                    return resolve()
                })
                .catch(function(e){
                    reject(e)
                })
        })
    })

Answer 1

云函数只允许 10 MB https://cloud.google.com/functions/quotas