Okio/Okhttp download file using BufferedSink and decode Base64 without having whole file in memory multiple times

Question

Got a bit of a problem atm. for my "inapp"-update im downloading the new base64 encoded .apk from my webspace. I have the functionality pretty much down, this is the code without decoding.

            public void onResponse(Call call, Response response) throws IOException {
            if(response.isSuccessful()){
                ResponseBody body = response.body();
                BufferedSource source = body.source();
                source.request(Long.MAX_VALUE);
                Buffer buffer = source.buffer();

                String rString = buffer.clone().readString(Charset.forName("UTF-8"));
                Log.i("Test: ", AppUtils.decodeBase64(rString));

                if(rString.equals("xxx")){
                    EventBus.getDefault().post(new KeyNotValid());
                    dispatcher.cancelAll();
                }else{
                    EventBus.getDefault().post(new SaveKey(apikey));
                    BufferedSink sink = Okio.buffer(Okio.sink(myFile));
                    sink.writeAll(source);
                    sink.flush();
                    sink.close();
                }
            }
        }

The Buffer/Log is not really necessary, just using it to check the response during testing.

How would i go about decoding the bytes before i write them to the sink? I tried doing it via. ByteString, but i couldn't find a way to write the decoded String back to a BufferedSource.

Most alternatives are pretty slow like reopening the file afterwards, reading the bytes into memory, decode and write them back.

Would really appreciate any help on this

cheers

Answer 1

You can already consume the response as an InputStream via ResponseBody.byteStream. You can decorate this stream with https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html and use it to read a stream of bytes and write it to the Sink for the file in chunks.

Answer 2

I know this answer arrives quite late and that Yuri's answer is technically correct, but I think the most idiomatic way to do that is to take advantage of the composition pattern promoted by Okio to create a Source that decodes from Base64 (or a Sink that encodes to Base64, if you need so).

Here's a little proof of concept (I'm sure it can be improved):

public class Base64Source implements Source {

    private Source delegate;
    private Base64.Decoder decoder; // Using Java 8 API, but it can be any library

    public Base64Source(Source delegate) {
        this(delegate, Base64.getDecoder());
    }

    public Base64Source(Source delegate, Base64.Decoder decoder) {
        this.delegate = delegate;
        this.decoder = decoder;
    }

    @Override
    public long read(Buffer sink, long byteCount) throws IOException {
        Buffer buffer = new Buffer();
        long actualRead = this.delegate.read(buffer, byteCount);
        if (actualRead == -1) {
            return -1;
        }

        byte[] encoded = buffer.readByteArray(actualRead);
        byte[] decoded = decoder.decode(encoded);
        sink.write(decoded);

        return decoded.length;
    }

    @Override
    public Timeout timeout() {
        return this.delegate.timeout();
    }

    @Override
    public void close() throws IOException {
        this.delegate.close();
    }
}

And here's how it can be used

BufferedSource source = Okio.buffer(new Base64Source(originalSource));
BufferedSink sink = ... // create sink
sink.writeAll(source);

// Don't forget to close the source/sink to flush and free resources
sink.close();
source.close();