有没有办法在用 Java 创建文件时将文件发送到 AWS S3 存储桶？

Question

I have incominig Data from a Database, which should be written to a CSV-file which again should be stored in a Amazon S3 Bucket. Im not allowed to use very much of my local storage (about 1GB). Is it possible to upload the incoming data as a CSV-file without using local storage or with just using that little i have? The file will have more than 10 GB.

Answer 1

It's pretty easy to do with AWS SDK, but the crucial thing is you need to know the file size before starting the upload

If you know, how big the file will be, then you can prepare your own InputStream and pass it to the S3 client like this:

public class DynamicUpload {

    public static void main(String[] args) {
        // Create S3 client
        AmazonS3 s3 = AmazonS3Client.builder().withRegion("eu-central-1").build();
        CsvStream stream = new CsvStream();
        // When providing InputStream, you must set content length
        ObjectMetadata obj = new ObjectMetadata();
        obj.setContentLength(stream.getSize());
        obj.setContentType("text/plain");
        // Pass created InputStream as a source
        s3.putObject(new PutObjectRequest("files.stirante.com", "stackOverflow.csv", stream, obj));
    }

    private static class CsvStream extends InputStream {

        private static DecimalFormat format = new DecimalFormat("00");
        // Target size for testing purposes
        private int size = 100000;
        // This is size of one row "XX;XX;XX\n"
        private int itemSize = 9;
        // Since we increment it at the very beginning, we set it to -1
        private int currentItemIndex = -1;
        // Current row, we're returning
        private byte[] currentItem = null;
        // Byte index in current row
        private int currentItemByteIndex = 0;

        /**
         * Returns size of the whole file
         */
        public int getSize() {
            return size * itemSize;
        }

        @Override
        public int read() throws IOException {
            // Every time read is called, we return another character from created earlier row
            currentItemByteIndex++;
            // If row is not initialized or earlier row was already fully returned, we create another row
            if (currentItem == null || currentItemByteIndex >= itemSize) {
                currentItemIndex++;
                // If we don't have another row, we throw end of file exception
                if (currentItemIndex == size) {
                    throw new EOFException();
                }
                // Format guarantees us, that in case of number smaller than 10, it will still return 2 characters (e.g. 02)
                String s = format.format(Math.random() * 99) + ";" +
                        format.format(Math.random() * 99) + ";" +
                        format.format(Math.random() * 99) + "\n";
                currentItem = s.getBytes();
                currentItemByteIndex = 0;
            }
            return currentItem[currentItemByteIndex];
        }
    }
}

Example generated file

Documentation: PutObjectRequest

Answer 2

I see 2 ways to achieve that:

1. Use multi part file uploading by handling 1 GB per part locally.
   See example .
2. Use cloud instance instead of local one (where you don't have 1 GB limitation) and use atomic upload (eg AmazonS3Client#putObject method)