boost::asio::async_read receiving EOF before receiving the complete Content-Length

Question

I am implementing an HTTP File download client using Boost ASIO. I am using the async_read operation. The problem I am facing is in the async_read I am receiving the EOF before receiving the full content (ie Content-Length), this is happening irrespective of the content size. Given is my read operation

void Http::ResumableAsyncDownload::read_content(const boost::system::error_code& err, size_t _size)
{
    try {
        if ( !err)
        {
            received_bytes += _size;
            if ( ofs_.is_open() ) {
            // Write all of the data that has been read so far.
            ofs_ << &response_;
        } else {
            ofs_.open(std::string(params_.tempdir + "/" + params_.partnumber).c_str());
            if ( ofs_.is_open() ) {
                ofs_ << &response_;
            } else {
                DEBUG_MSG("Error while opening file to store downloaded data. File Path = %s\n", std::string(params_.tempdir + "/" + params_.partnumber).c_str());
                std::cout << ("Unable to open local file for storing downloaded data");
            }
        }
        // Continue reading remaining data until EOF.
        boost::asio::async_read(*ssocket_, response_,
                            boost::asio::transfer_at_least(1),
                            boost::bind(&ResumableAsyncDownload::read_content, this, _1, _2)
                        );
        }
        else if (err != boost::asio::error::eof)
        {
            DEBUG_MSG("[NET] : Exception in ResumableAsyncDownload in read_content : %s\n", err.message().c_str());
            std::cout << ("Asynchronous File Download Error: " + err.message());
        }

        if(err == boost::asio::error::eof)
        {
            std::cout << "[RESPONSE] : EOF: We are not breaking connection\n";
            ssocket_->shutdown();
            delete ssocket_;
            ssocket_ = NULL;
            delete ctx;
            ctx      = NULL;

            if ( (content_length != received_bytes) && !(params_.get_size) ) {
                std::cout << "Failed to receive complete data packet. Content Length = " << content_length << " Received Bytes = " << received_bytes << std::endl;
            // ofs_.clear();
            }
        }
    } catch ( std::exception &ex ) {
        std::cout << "We have an exception. Exception = " << std::string(ex.what()) << std::endl;
    }
}

So for example, the Content-Length is say 292309324, but I will receive EOF before 292309324.

To overcome this problem I have implemented the Chunked download using the HTTP Range header, but in that case for every chunk I request I receive less than the requested chunk, then I re-calculate the next range, it works before the last chunk. I never receive the last chunk and usually the situation is (ie)

Range for last chunk 227376464-227376641/227376641 
Requested Bytes = 178

Response Headers

X-Powered-By: Undertow/1
Content-Range: bytes 227376464-227376641/227376641
Server: WildFly/9
Content-Length: 178
Accept-Ranges: bytes
OperationId: 4a847024-2348-42bd-af7d-3638e41cba4f
Date: Thu, 17 Aug 2017 11:41:18 GMT
Set-Cookie: SERVERID=04-84FRD2128G0US; path=/
Cache-control: private

As you can see, the server is responding with good range of the last chunk but in the read_content is giving EOF.

So in both approaches read_content is not reading the complete data and giving EOF. As I understand EOF is the socket closed by the server and can also cause short-read but isn't my chunked download solution. Shouldn't I receive the last chunked packet in full.

Any thoughts on whats going wrong ? Please also note that I am calling a custom API to download the file but I am seeing the same issue even if I download from some public link (ie http://mirror.pnl.gov/releases/16.04/ubuntu-16.04.3-desktop-amd64.iso ), so I do not think the issue is at my server side. Also note that I do not see this issue if I use the synchronous version of boost::asio::async_read (ie boost::asio::read). I am using Boost version 1.55 compiled for ARM.

Answer 1

I recommend adding a total byte counter to your Http class size_t bytes_transferred_total_; and ditching err == boost::asio::error::eof . You know the total size as that is part of the original HTTP header Content-Length: <total_body_size> which you've parsed. The amended code would look something like this:

...
if (!err) {
  received_bytes += _size;
  bytes_transferred_total_ += _size;
  ...
} else {
  ...
}
boost::asio::async_read(
    *ssocket_, response_, boost::asio::transfer_at_least(1),
    boost::bind(&ResumableAsyncDownload::read_content, this, _1, _2));
// continue reading until all data has been received
if (bytes_transferred_total_ >= content_length_from_header_) {
  // you've received it all
}

In initializing content_length_from_header_ make sure you understand that it represents the size of the body itself, excluding the header.

Footnote: Instead of boost::bind or std::bind consider using lambdas as they are generally more performant, eg allowing compilers to inline them. Unless, of course, you need to exploit bind 's dynamic nature.