云存储中的 Memory 问题 Function

Question

I have deployed a storage trigger cloud function that needs more memory. While deploying the GCF, I have deployed in the following manner with the appropriate flags.

gcloud functions deploy GCF_name--runtime python37 --trigger-resource bucket_name --trigger-event google.storage.object.finalize --timeout 540s --memory 8192MB

But I observed in the google cloud console, the memory utilization map is not going beyond 2GB. And in the logs I am getting this error, Function execution took 34566 ms, finished with status: 'connection error' which happens because of memory shortage. Can I get some help on this.

Edited

The application uploads text files to the storage that contains certain number of samples. Each file is read when it is uploaded to the storage and the data appended to a pre existing file. The total number of samples will be maximum of 75600002. That's why I need 8GB data. Its giving the connection error while appending the data to the file.

def write_to_file(filename,data,write_meta = False,metadata = []):
    file1 = open('/tmp/'+ filename,"a+")
    if write_meta:
        file1.write(":".join(metadata))
        file1.write('\n')
    file1.write(",".join(data.astype(str)))
    file1.close()

The memory utilisation map was the same after every upload.

Answer 1

You are writing a file to /tmp which is an in-memory filesystem. So start by deleting that file when you finish uploading it. In fact those:

Files that you write consume memory available to your function, and sometimes persist between invocations. Failing to explicitly delete these files may eventually lead to an out-of-memory error and a subsequent cold start.

Ref: https://cloud.google.com/functions/docs/bestpractices/tips#always_delete_temporary_files