ssl_read(ssl, buffer, sizeof(buffer)) does not read json return correctly

Question

I am working with embedded linux, C, and openSSL. I am hitting my REST server with a GET and expecting a json object response. The server gets the correct REST Get and handles it correctly. However, for some reason the bytes = ssl_read() is not properly receiving the json object. I know the server is sending the correct json object because I can hit the server with a Postman or Curl script and I get the correct json object. Also the logger(debug) shows that the server is responding correctly with the correct json object. If I modify the server endpoint code to respond with a string and return a jsonObject.ToString() I do get the correct response but it is a string. How come I am not getting a good json object? Here is an abbreviated code listing,

 // Define some parameters
 int sockfd, bytes_read;
 struct sockaddr_in dest;
 struct hostent *hostent;
 unsigned short svrport = 8443;

 char hdr[4000];
 char buffer[4000];

 // Initialise Crypto library
 ERR_load_crypto_strings();
 OpenSSL_add_all_algorithms();
 OPENSSL_config(NULL);

 // Create GET status HTTP Client Message String
 bzero(hdr, sizeof(hdr));
 strcpy(hdr, "GET /status HTTP/1.1\r\n");
 strcat(hdr, "Host: xxx.xxx.xxx.xxx\r\n");
 strcat(hdr, "X-BMID-ID: 1234-5678\r\n");
 strcat(hdr, "Authorization: ");
 strcat(hdr, sttb64e);
 strcat(hdr, "\r\n");
 strcat(hdr, "Accept: application/json\r\n");
 strcat(hdr, "Content-Type: application/json\r\n");
 strcat(hdr, "Connection: close\r\n");
 strcat(hdr, "\r\n");  

 // Initialize server address/port struct
 bzero(&dest, sizeof(dest));
 dest.sin_family = AF_INET;
 dest.sin_port = htons(svrport);
 dest.sin_addr.s_addr = inet_addr("xxx.xxx.xxx.xxx");

 if ( dest.sin_addr.s_addr == INADDR_NONE ) {
        printf("Incorrect Address Expression\n");
        return 0;
 }

 // Connect Socket
 if ( connect(sockfd, (struct sockaddr*)&dest, sizeof(dest)) != 0 ) {
      printf("Socket Connection Failed\n");
      close(sockfd);
      return 0;
 }

 // Now initialize SSL Engine
 SSL_METHOD const *method;
 SSL_CTX *ctx;
 SSL *ssl;

 SSL_library_init();
 OpenSSL_add_ssl_algorithms();
 SSL_load_error_strings();
 method = TLSv1_2_client_method();
 ctx = SSL_CTX_new(method);
 SSL_CTX_set_options(ctx, SSL_OP_NO_SSLv2);

 // Make sure SSL Engine is ok
 if (ctx == NULL) {
    printf("SSL Engine initialization failed\n");
    ERR_print_errors_fp(stderr);
    close(sockfd);
    return 0;
 }

 // Now create the SSL connection state
 ssl = SSL_new(ctx);
 SSL_set_fd(ssl, sockfd);

 // Check to see if the SSL socket was connected
if (SSL_connect(ssl) == FAIL) {
    printf("SSL socket failed to connect\n");
    return 0;
 }

 // Now Send Secure Message
 bzero(buffer, sizeof(buffer));
 SSL_write(ssl, hdr, strlen(hdr));
 bytes = SSL_read(ssl, buffer, sizeof(buffer));
 buffer[bytes] = 0;
 if (verbose) {
    printf("Received Header:\n");
    printf("%s\n", buffer);
 }
 bzero(buffer, sizeof(buffer));
 bytes = SSL_read(ssl, buffer, sizeof(buffer)); <=== **** Error is Here ****
 buffer[bytes] = 0;
 printf("Received Body:%s\n\n", buffer);
 printf("Buffer Length = %i\n\n", bytes);
 SSL_free(ssl);

 // Shut down connection
 close(sockfd);
 SSL_CTX_free(ctx);

The "Received Body: just has 2 characters, always numbers, and the Buffer Length is 4 bytes. The real json object looks like this,

 {"status":"OK","shadowRequest":"0","serviceCount":1}

Like I said, this code works all works fine except it does not get the correct json object stored in the buffer. The ssl_read() in question is locate at the end of the listing and marked <=== **** Error is Here ****. Any ideas?

Answer 1

You need to read about transfer encoding "chunked" which is not simply plain text. You ask for sizeof(buffer) but the correct way is to read until you find a 0 that comes from the server as text in a single line ( as far as I can remember ) meaning the chunk s are complete.

I created a simple HTTP library for a project I am working on right now. And I analyze the HTTP headers first interested in primarily 3 things

1. The Content-Length header: if present simply get the value and read that from the server in a single read , although that means a read loop with OpenSSL it's not that simple.

2. The Transfer-Encoding header: Here you can determine how to read the data if the content-length was not present.

3. The Content-Encoding header: Used to determine if the content is compressed for example with gzip .

In my particular case this is enough. But this is just a tiny part of the protocol and it's not suitable in general. Although, maybe it could work for you.

So you need to get all the headers from the server first, read every line until an empty line appears. After you have the headers, you can then use the information to decide how to read the response body. Basically, you want to implement a simple HTTP client, for which at least the very basic things of the HTTP protocol should be implemented.

It is a lot of work, and I would guess that if you can install OpenSSL it's highly likely that you can use libcurl but I am not sure since I have never worked with embedded linux.

NOTE : If you understand how c strings work, you would not use strcat() like that.

Answer 2

Two things:

1. Don't claim HTTP/1.1 support unless you're going to implement every requirement of the HTTP 1.1 specification.

2. You can't assume that your first call to SSL_read will get you the headers and your next one will get you the body. You have to actually implement the HTTP protocol. SSL doesn't understand HTTP and has no idea what a header or body is or how to read them.

Honestly, your level of protocol knowledge suggests that you are way out of your depth here. You really should find an existing HTTP client implementation that meets your needs rather than trying to roll your own. Even if it happens to work every time you test it, there are a remarkable number of subtle mistakes that it is possible to make and every person who is new to this makes pretty much every one of them.

For example:

 bytes = SSL_read(ssl, buffer, sizeof(buffer));
 buffer[bytes] = 0;

What if bytes == sizeof(buffer) after the first line? Your second line will write past the end of the buffer.