What is the best practice to delete multiple objects(files and folders) from google cloud storage using node.js

Question

I want to delete multiple objects from my google cloud storage, which means delete folders and files in combination. There is an example in the official documentation:

bucket.deleteFiles({ prefix: 'image-' }, callback);

This will delete all files in the images directory. But what if I want to delete 10 000 files in this directory? My demo code is this:

const _deleteItems = async (gcsClient, bucketName, allItems) => {
    try {
        for (let item of allItems) {
            await gcsClient.bucket(bucketName).deleteFiles({ prefix: getFilePath(item) }, function(err) {});
        }
    } catch (err) {
        throw new Error('Could not delete item');
    }
};

Is there a better way to do this? What if someone else in the meantime upload new file in the folder that i am going to delete? That file also will be deleted in this case.

Answer 1

• Is there a better way to do this?

  Here you may find the best practices regarding object deletion and bulk object deletion which applies to your use case.

• What if someone else in the meantime upload new file in the folder that i am going to delete? That file also will be deleted in this case.

  In this case you need to use Object Lifecycle Management and set a rule to protect files with a certain "age". You can also use Object versioning which "protects your Cloud Storage data from being overwritten or accidentally deleted" but please keep in mind the following:

  • "Enabling Object Versioning increases storage costs"

  • "When using Object Lifecycle Management in buckets with object versioning enabled, deleting the live version of an object creates a noncurrent version, while deleting a noncurrent version deletes that version permanently."