如何在Java中从S3读取Snappy压缩文件

Question

Currently we are running MapReduce job in Hadoop in which the output is compressed into SnappyCompression. Then we are moving the output file to S3. Now I want to read the Compressed file from S3 through Java.

Answer 1

I found the answer to read snappy compressed file from S3. First you should get the object content from S3. And then decompress the file.

    S3Object s3object = s3Client.getObject(new GetObjectRequest(bucketName,Path));
    InputStream inContent = s3object.getObjectContent();
    CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(SnappyCodec.class, new Configuration());
    InputStream inStream = codec.createInputStream(new BufferedInputStream(inContent));
    InputStreamReader  inRead = new InputStreamReader(inStream);
    BufferedReader br = new BufferedReader(inRead);
    String line=null;
    while ((line = br.readLine()) != null){
        system.out.println(line);
    }