com.google.api.client.googleapis.json.GoogleJsonResponseException while reading from google bucket

Question

I am trying to list all the files from one directory in google bucket and reading data from each file, for some buckets it is working but for one bucket it is throwing error

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 500 Internal Server Error { "code" : 500, "errors" : [ { "domain" : "global", "message" : "Internal Error", "reason" : "responseTooLarge" } ], "message" : "Internal Error"

stacktrace: Exception in thread "main" com.google.cloud.storage.StorageException: Internal Error at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:220) at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:346) at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:299) at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:296) at com.google.api.gax.retrying.DirectRetryingExecutor.submit( at com.google.cloud.RetryHelper.run(RetryHelper.java:74) at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51) at com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:295) at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:262) at WriteToGs.main(WriteToGs.java:22)

import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;


public class WriteToGs {
    static PGPContainer pgpContainer = new PGPContainer();
    final static Storage storage = StorageOptions.getDefaultInstance().getService();
    public static void main(String[] args) throws Exception {
        final DateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmm");
        final Date date = new Date();
        final String destinationDirectory=args[2];
        final String flowname=args[3];

        final String num_threads=args[4];

        Iterable<Blob> blobs = storage.list(args[0], Storage.BlobListOption.prefix(args[1])).iterateAll();
        ExecutorService executorService = Executors.newFixedThreadPool(Integer.parseInt(num_threads));
        for (Blob blob : blobs) {
            System.out.println("Blob name:"+blob.getName());
            final String destinationFilename=blob.getName();
            final Blob contentBlob=blob;

            executorService.execute(new Runnable() {
                public void run() {
                    String fileContent = new String(contentBlob.getContent());
                    BlobId newblobId = BlobId.of(destinationDirectory,"xyz/fds" + flowname + "/"
                            + dateFormat.format(date) + "/" + destinationFilename.replace(".txt",".pgp"));
                    BlobInfo blobInfo = BlobInfo.newBuilder(newblobId).setContentType("text/plain").build();
                    try {

                        System.out.println("Blob name deleted:"+contentBlob.getName());
                        storage.delete(contentBlob.getBlobId());
                    } catch (Exception e) {
                        e.printStackTrace();
                    }




                }
            });
        }
        executorService.shutdown();
    }

}

Answer 1

Well, assuming that the line 22 (which is where the error came from) is the one:

System.out.println("Blob name:"+blob.getName());

Probably, the property “name” inside blob is large enough to break the application. But I'm almost sure this is not the line where the application is being broken.

Based on a similar question, GoogleJsonResponseException: 500 Internal Server Error : Response Too Large To Return , I suggest you to create a bug, since this could be a problem directly related with Google libraries.