Boost asio ssl socket doesn't receive complete data different behaviour of tcp socket and ssl socket

Question

I'm using

boost::asio::ssl::stream<boost::asio::ip::tcp::socket>

and

boost::asio::ip::tcp::socket

in a template method to receive data (HTTP(S)-POST-Request with data (multi-part)) from a Webbrowser (Firefox).

template<class SocketType>
void handleHTTPSRequest(SocketType& socket, ...........)

My problem is, that the behaviour of this two sockets is different. In a loop:

av = socketAvailable(socket);
while(av == 0 || timeout...)
{
    av = socketAvailable(socket);
}
if(av > 2048)
{
    av = 2048;
}
recv = socket.read_some(boost::asio::buffer(buffer, av));
if(recv <= 0) // ignoring that recv is 0 doesn't help receiving the complete data.
{
    break; //and use data
}

The data is not complete when I use

boost::asio::ssl::stream<boost::asio::ip::tcp::socket>

instead of

boost::asio::ip::tcp::socket

which works for me.

Answer 1

You should make the code self-contained. We can't tell what socketAvailable is doing.

However, in general just reading "whatever is available" is identical to "read_some" (given enough buffer) and is never a good way to receive the full request, because the organization and timing of the lower-layer packets are not something you can rely on.

You need to do read_until or read with "unbounded" buffer to read the full request.

Demo Using Beast

Since you're using Boost, consider using Beast to read the request:

template <typename Stream>
void handle_request(Stream& s) {
    http::request<http::string_body> request;
    beast::flat_buffer buf;
    read(s, buf, request);

    std::cout << "The full request headers: " << request.base() << "\n";
    std::cout << "Body size is: " << request.body().length() << "\n";
    std::cout << "Trailing unused buffer data: " << buf.size() << "\n";

    // write body to a file
    std::ofstream("body.txt", std::ios::binary)
        << request.body();

    // be polite and respond
    auto res
        = http::response<http::string_body>(http::status::ok, 11, "Thanks\r\n");
    write(s, res);
}

This will work with any AsyncReadStream - so a plain TCP socket as well as an SSL stream on top of one.

Let's add a test program that can use SSL optionally:

int main(int argc, char** argv) {
    std::set<std::string_view> const args(argv+1, argv+argc);
    bool const use_ssl = args.count("ssl");
    bool const single_request = args.count("single");
    net::io_context io;
    tcp::acceptor acc(io, {{}, 8099});

    while(true) {
        tcp::socket conn = acc.accept();

        if (!use_ssl) {
            handle_request(conn);
        } else {
            using Stream = beast::ssl_stream<tcp::socket>;
            using Ctx = ssl::context;
            Ctx ctx(Ctx::method::sslv23);
            ctx.use_certificate_file("server.pem", Ctx::pem);
            ctx.set_password_callback( [](size_t, Ctx::password_purpose) { return "test"; });
            ctx.use_rsa_private_key_file("server.pem", Ctx::pem);

            Stream s(std::move(conn), ctx);
            s.handshake(Stream::server);

            handle_request(s);
            s.shutdown();
        }

        if (single_request)
            break;
    }
}

Now, we can run it either in plain HTTP mode or HTTPS:

./sotest               # HTTP
./sotest ssl           # HTTPS

Testing

Using a simple curl to send a multipart:

curl -X POST -k -F 'upload=@main.cpp' http://localhost:8099
curl -X POST -k -F 'upload=@main.cpp' https://localhost:8099

Both work as advertised, and curl prints the server response:

HTTP/1.1 200 OK

Thanks

Live Demo

Live On Coliru

#include <boost/beast.hpp>
#include <boost/beast/http.hpp>
#include <boost/beast/ssl.hpp>
// for debug output only:
#include <iostream>
#include <fstream>
#include <set>

namespace beast = boost::beast;
namespace http = beast::http;
namespace net = boost::asio;
namespace ssl = net::ssl;
using net::ip::tcp;
using boost::system::error_code;

template <typename Stream>
void handle_request(Stream& s) {
    http::request<http::string_body> request;
    beast::flat_buffer buf;
    read(s, buf, request);

    std::cout << "The full request headers: " << request.base() << "\n";
    std::cout << "Body size is: " << request.body().length() << "\n";
    std::cout << "Trailing unused buffer data: " << buf.size()              << "\n";

    // write body to a file
    std::ofstream("body.txt", std::ios::binary)
        << request.body();

    // be polite and respond
    auto res
        = http::response<http::string_body>(http::status::ok, 11, "Thanks\r\n");
    write(s, res);
}

int main(int argc, char** argv) {
    std::set<std::string_view> const args(argv+1, argv+argc);
    bool const use_ssl = args.count("ssl");
    bool const single_request = args.count("single");
    net::io_context io;
    tcp::acceptor acc(io, {{}, 8099});

    while(true) {
        tcp::socket conn = acc.accept();

        if (!use_ssl) {
            handle_request(conn);
        } else {
            using Stream = beast::ssl_stream<tcp::socket>;
            using Ctx = ssl::context;
            Ctx ctx(Ctx::method::sslv23);
            ctx.use_certificate_file("server.pem", Ctx::pem);
            ctx.set_password_callback( [](size_t, Ctx::password_purpose) { return "test"; });
            ctx.use_rsa_private_key_file("server.pem", Ctx::pem);

            Stream s(std::move(conn), ctx);
            s.handshake(Stream::server);

            handle_request(s);
            s.shutdown();
        }

        if (single_request)
            break;
    }
}

With the commands

g++ -std=c++17 -O2 -Wall -pedantic -pthread main.cpp -lssl -lcrypto -o sotest
(./sotest ssl single &
sleep 1
curl -s -i -X POST -k -F 'upload=@main.cpp' https://127.0.0.1:8099
wait)

The server prints

The full request headers: POST / HTTP/1.1
Host: 127.0.0.1:8099
User-Agent: curl/7.47.0
Accept: */*
Content-Length: 2199
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------cdd7b913a4ef2f57

Body size is: 2199
Trailing unused buffer data: 0

And curl prints:

HTTP/1.1 200 OK
Thanks

Note This uses server.pem as included in the Asio examples for certificate/private key loading

Answer 2

Thanks for the detailed solution using beast.

I have figured out another solution for my use case:

Missing code for socketAvailable

std::size_t socketAvailable(boost::asio::ssl::stream<boost::asio::ip::tcp::socket>& socket)
{
    return socket.next_layer().available();
}

std::size_t socketAvailable(boost::asio::ip::tcp::socket& socket)
{
    return socket.available();
}

My solution is working when the content-length is set in the request head:

knownreceivesize = contentLength + requestHeadSize (untill (including) \r\n\r\n)

transferSize is the number of bytes already been received from the request.

av = socketAvailable(server);

if(av > 2048)
{
    av = 2048;
}
if (knownreceivesize > 0 && av < knownreceivesize - transferSize)
{
    av = knownreceivesize-transferSize;
    if (av > 2048)
    {
        av = 2048;
    }
}