C HTTP 客户端从响应中剥离标头

Question

So, I'm writing this simple HTTP client in C and I seem to be stuck on this problem - how do I strip the HTTP headers from the response? After all, if I get a binary file I can't just write the headers out to my output file. I can't seem to go in once the data is already written to a file because linux screams when you try to even view the first few lines of a binary file, even if you know they're just text HTTP headers.

Now, here's the rub (well, I suppose the whole thing is a rub). Sometimes the whole header doesn't even in come in on the first response packet, so I can't even guarantee that we'll have the whole header in our first iteration (that is, iteration of receiving an HTTP response. We're using recv() , here), which means I need to somehow... well, I don't even know. I can't seem to mess with the data once it's already written to disk, so I need to deal with it as it's coming in, but we can't be sure how it's going to come in, and even if we were sure, strtok() is a nightmare to use.

I guess I'm just hoping someone out there has a better idea. Here's the relevant code. This is really stripped down, I'm going for MCVE, of course. Also, you can just assume that socket_file_descriptor is already instantiated and get_request contains the text of our GET request. Here is it:

FILE* fp = fopen("output", "wb"); // Open the file for writing
char buf[MAXDATASIZE]; // The buffer
size_t numbytes; // For the size of the response

/*
 * Do all the socket programming stuff to get the socket file descriptor that we need
 * ...
 * ...
*/

send(socket_file_descriptor, get_request, strlen(get_request), 0); // Send the HTTP GET request

while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
    /* I either need to do something here, to deal with getting rid of the headers before writing to file */
    fwrite(buf, 1, numbytes, fp); // Write to file
    memset(buf, 0, MAXDATASIZE); // This just resets the buffer to make room for the next packet
}
close(s);
fclose(fp);
/* Or I need to do something here, to strip the file of its headers after it's been written to disk */

So, I thought about doing something like this. The only thing we know for sure is that the header is going to end in \\r\\n\\r\\n (two carriage returns). So we can use that. This doesn't really work, but hopefully you can figure out where I'm trying to go with it (comments from above removed):

FILE* fp = fopen("output", "wb");
char buf[MAXDATASIZE];
size_t numbytes;
int header_found = 0; // Add a flag, here

/* ...
 * ...
*/

send(socket_file_descriptor, get_request, strlen(get_request), 0);

while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
    if (header_found == 1) { // So this won't happen our first pass through
        fwrite(buf, 1, numbytes, fp);
        memset(buf, 0, MAXDATASIZE);
    }
    else { // This will happen our first pass through, maybe our second or third, the header doesn't always come in in full on the first packet
        /* And this is where I'm stuck.
         * I'm thinking about using strtok() to parse through the lines, but....
         * well I just can't figure it out. I'm hoping someone can at least point
         * me in the right direction.
         *
         * The point here would be to somehow determine when we've seen two carriage returns
         * in a row and then mark header_found as 1. But even if we DID manage to find the
         * two carriage returns, we still need to write the remaining data from this packet to 
         * the file before moving on to the next iteration, but WITHOUT including the
         * header information.
        */
    }
}
close(s);
fclose(fp);

I've been staring at this code for three days straight and am slowly losing my mind, so I really appreciate any insight anyone is able to provide. To generalize the problem, I guess this really comes down to me just not understanding how to do text parsing in C.

Answer 1

The second self-answer is better than the first one , but it still could be made much simpler:

const char* pattern = "\r\n\r\n";
const char* patp = pattern;
while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
    for (int i = 0; i < numbytes; i++) {
        if (*patp == 0) {
            fwrite(buf + i, 1, numbytes - i, fp);
            break;
        }
        else if (buf[i] == *patp) ++patp;
        else patp = pattern;
    }
    /* This memset isn't really necessary */
    memset(buf, 0, MAXDATASIZE);
}

That looks like a general solution, but it's not really: there are values for pattern for which it might fail to see a terminator under particular circumstances. But this particular pattern is not problematic. You might want to think about what sort of pattern would cause a problem before taking a look at the more general solution .

Answer 2

So, I know this is not the most elegant way to go about this, but... I did get it. For anyone who finds this question and is curious about at least an answer, here it is:

int count = 0;
int firstr_found = 0;
int firstn_found = 0;
int secondr_found = 0;
int secondn_found = 0;
FILE* fp = fopen("output", "wb");
char buf[MAXDATASIZE];
size_t numbytes;
int header_found = 0;

/* ...
 * ...
*/

send(socket_file_descriptor, get_request, strlen(get_request), 0);

while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
    if (header_found == 1) {
        fwrite(buf, 1, numbytes, fp);
    }
    else {
        // These buf[i]'s are going to return as integers (ASCII)
        // \r is 13 and \n is 10, so we're looking for 13 10 13 10
        // This also needs to be agnostic of which packet # we're on; sometimes the header is split up.
        for (int i = 0; i < numbytes; i++) {
            if (firstr_found == 1 && firstn_found == 1 && secondr_found == 1 && secondn_found == 1) { // WE FOUND IT!
                header_found = 1;
                // We want to skip the parts of the buffer we've already looked at, that's header, and our numbytes will be decreased by that many
                fwrite(buf + i, 1, numbytes - i, fp);
                break;
            }
            
            if (buf[i] == 13 && firstr_found == 0) { // We found our first \r, mark it and move on to next iteration
                firstr_found = 1;
                continue; 
            }
            if (buf[i] == 10 && firstr_found == 1 && firstn_found == 0) { // We found our first \n, mark it and move on
                firstn_found = 1;
                continue; 
            }
            else if (buf[i] != 13 && buf[i] != 10) { // Think about the second r, it'll ignore the first if, but fail on the second if, but we don't want to jump into this else block
                firstr_found = 0;
                firstn_found = 0;
                continue;
            }
            if (buf[i] == 13 && firstr_found == 1 && firstn_found == 1 && secondr_found == 0) {
                secondr_found = 1;
                continue;
            }
            else if (buf[i] != 10) {
                firstr_found = 0;
                firstn_found = 0;
                secondr_found = 0;
                continue;
            }
            if(buf[i] ==  10 && firstr_found == 1 && firstn_found == 1 && secondr_found == 1 && secondn_found == 0) {
                secondn_found = 1;
                continue;
            }
        }
    }
    memset(buf, 0, MAXDATASIZE);
    count++;
}
close(s);
fclose(fp);

Answer 3

Adding another answer because, well I suppose I think I'm clever. Thanks to @tadman for the idea of a counter. Look here (I'm going to shave off a lot of the bloat and just do the while loop, if you've looked at my other code blocks you should be able to see what I mean here) ...

/* ...
 * ...
*/
int consec_success = 0;
while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
    if (header_found == 1) {
        fwrite(buf, 1, numbytes, fp);
    }
    else {
        for (int i = 0; i < numbytes; i++) {
            if (consec_success == 4) {
                header_found = 1;
                fwrite(buf + i, 1, numbytes - i, fp);
                break;
            }
            
            if (buf[i] == 13 && consec_success % 2 == 0) {
                consec_success++;
            }
            else if (buf[i] == 10 && consec_success % 2 == 1) {
                consec_success++;
            }
            else {
                consec_success = 0;
            }
        }
    }
    memset(buf, 0, MAXDATASIZE);
}
/* ...
 * ...
*/