How to upload multiple files to Google Cloud Storage in a single call using Java API?

Question

We want to upload multiple files to Google Cloud Storage. Currently, we are uploading one by one using the Google Java API. The code is below:

public void uploadFile(File srcFile,String bucketName, String destPath) throws IOException {
      BlobId blobId = BlobId.of(bucketName, srcFile.getName());
      BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();

  long startTime = System.currentTimeMillis();

 // Blob blob = storage.create(blobInfo,new FileInputStream(srcFile));

  try (WriteChannel writer = storage.writer(blobInfo)) {
    try (FileInputStream in = new FileInputStream(srcFile)){


        byte[] buffer = new byte[1024 * 1024 * 100] ;

        writer.setChunkSize(buffer.length);

        int readSize = 0;

        while((readSize = in.read(buffer)) > 0) {
             writer.write(ByteBuffer.wrap(buffer, 0, readSize));
        }

        long endTime = System.currentTimeMillis();

      double writeTime = (double)(endTime - startTime) / 1000;

      System.out.println("File write time : " + writeTime);

    }
  }


}

Our application wants to upload multiple files at a time. I tried to find a method to upload multiple files in the Java API, but could not find the any method to upload multiple files using a single call.

If I loop and upload using multiple calls, it is adding a huge network overhead and performance is very slow, which the application cannot afford.

My questions are:

1. Is it possible to upload multiple files via a single call, either using the Java API or REST API?
2. Could you please provide an example?

Answer 1

Neither the google-cloud-java library nor the underlying JSON API has an API call to upload multiple files, but you could accomplish what you're trying to do by spawning multiple threads and having each thread upload a file. The gsutil -m option does it this way.