How do I stream the body of an HTTP request using Java

Question

I have an InputStream and the size of the data that is going to come out (Response of HTTP request). For space complexity reasons I can't read the whole thing. What I want is to send the data directly into a new request body. I've tried doing this with OkHttp but I can't get it to work. I'm not aware of any other HTTP clients that can do this.

If at all possible I'd like to avoid messing around with Socket . Any advice?

Edit: Added restriction is that the solution must work with Java 8

Answer 1

I believe the new HttpClient standardized in Java 11 should let you do this. It uses the Flow API (reactive streams), and you can provide a BodyHandler / BodySubscriber that will request/receive the bytes as they come. The HttpClient also lets you specifies a BodyPublisher when you send out a request. So it should only be a matter of tying the subscription forwarded by the request publisher to its subscribers, to the subscription handed out to BodySubscriber by the Http stack, and then having the BodySubscriber 's onNext (etc..) invoke the Publisher 's subscriber corresponding methods. Note that this is an academic description: I haven't actually tried to implement it. It might require some thinking and some tricks to set up the subscription link but I believe it should work.

Make sure that your implementation of BodySubscriber / BodyPublisher adhere to the reactive streams semantics however - and that they do not block in the callbacks.

https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html https://openjdk.java.net/groups/net/httpclient/intro.html

On second thought maybe that's not what you're asking: if you already have an InputStream it's even simpler: just use BodyPublishers.ofInputStream when you send your request.

https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpRequest.BodyPublishers.html#ofInputStream(java.util.function.Supplier)

Answer 2

I would add that if the request is of sufficient size that you are wanting to implement streaming it, it is likely that the receiving service would also want to implement streaming as well.

It also true that while the older existing HttpUrlConnection class, or anything that provides access to the Input and Output streams associated with a given Url connection could support streaming, it is also likely that you would have to code an inordinate amount of support for newer security concerns such as OPTIONS.

Answer 3

You can do something like this:

    File file = new File("path_to_file"); // specify path to the large file you want to upload

    InputStream inputStream = new FileInputStream(file); // Create input stream out of the file. This will be used to stream data directly to request body

    HttpURLConnection connection = (HttpURLConnection) new URL("remote_url").openConnection(); // Open connection. This will not send eny request until explicitly asked. 
    connection.setDoOutput(true); // This will set http method to POST
    connection.setFixedLengthStreamingMode(file.length()); // Define the length of the request body size

    OutputStream output = connection.getOutputStream(); // This is the output stream of the request body. It will be used to stream file bytes into.

    var bytes = inputStream.readNBytes(1024); // Read the first 1024 bytes from the file

    // Following while loop will read chunks of 1024 bytes from the file and write them into the request body output stream.
    // Note the use of the flush function. It will send the current content of the output stream to the remote server. 
    while (bytes.length > 0) {
        output.write(bytes);
        output.flush();
        bytes = inputStream.readNBytes(1024);
    }


    connection.connect(); // This finishes request and returns response

I used this for integration with Flussonic and it does the job.

You can use this answer for further research: https://stackoverflow.com/a/2793153/11676066