Libcurl C++: Non-blocking way to send requests to a single URL

Question

Goal: To send requests to the same URL without having to wait for the request-sending function to finish executing.

Currently when I send a request to a URL, I have to wait around 10 ms for the server's response before sending another request using the same function. The aim is to detect changes on a webpage slightly faster than the program currently is doing, so for the WHILE loop to behave in a non-blocking manner.

Question: Using libcurl C++, if I have a WHILE loop that calls a function to send a request to a URL, how can I avoid waiting for the function to finish executing before sending another request to the SAME URL ?

Note: I have been researching libcurl's multi-interface but I am struggling to determine if this interface is more suited to parallel requests to multiple URLs rather than sending requests to the same URL without having to wait for the function to finish executing each time. I have tried the following and looked at these resources:

• an attempt at multi-threading a C program using libcurl requests
• How to do curl_multi_perform() asynchronously in C++?
• http://www.godpatterns.com/2011/09/asynchronous-non-blocking-curl-multi.html
• https://curl.se/libcurl/c/multi-single.html
• https://curl.se/libcurl/c/multi-poll.html

Here is my attempt at sending a request to one URL, but I have to wait for the request() function to finish and return a response code before sending the same request again.

#include <vector>
#include <iostream>
#include <curl/curl.h>

size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata) {

        std::vector<char> *response = reinterpret_cast<std::vector<char> *>(userdata);
        response->insert(response->end(), ptr, ptr+nmemb);
        return nmemb;
}

long request(CURL *curl, const std::string &url) {

        std::vector<char> response;
        long response_code;

        curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
        curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response_code);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);

        auto res = curl_easy_perform(curl);
     
        // ...
        // Print variable "response"
        // ...

        return response_code;
}

int main() {
        curl_global_init(CURL_GLOBAL_ALL);
        CURL *curl = curl_easy_init();

        while (true) {
                // blocking: request() must complete before executing again
                long response_code = request(curl, "https://example.com");
                // ...
                // Some condition breaks loop
        }

        curl_easy_cleanup(curl);
        curl_global_cleanup();
        return 0;
}

I'm at a point where I have tried to understand the multi-interface documentation as best as possible, but still struggle to fully understand it / determine if it's actually suited to my particular problem. Apologies if this question appears to have not provided enough of my own research, but there are gaps in my libcurl knowledge I'm struggling to fill.

I'd appreciate it if anyone could suggest / explain ways in which I can modify my single libcurl example above to behave in a non-blocking manner.

---

EDIT:

From libcurl's C implemented example called "multi-poll", when I run the below program the URL's content is printed, but because it only prints once despite the WHILE (1) loop I'm confused as to whether or not it is sending repeated non-blocking requests to the URL (which is the aim), or just one request and is waiting on some other change/event?

#include <stdio.h>
#include <string.h>

/* somewhat unix-specific */
#include <sys/time.h>
#include <unistd.h>

/* curl stuff */
#include <curl/curl.h>

int main(void)
{
        CURL *http_handle;
        CURLM *multi_handle;
        int still_running = 1; /* keep number of running handles */

        curl_global_init(CURL_GLOBAL_DEFAULT);

        http_handle = curl_easy_init();

        curl_easy_setopt(http_handle, CURLOPT_URL, "https://example.com");

        multi_handle = curl_multi_init();

        curl_multi_add_handle(multi_handle, http_handle);

        while (1) {
                CURLMcode mc; /* curl_multi_poll() return code */
                int numfds;

                /* we start some action by calling perform right away */
                mc = curl_multi_perform(multi_handle, &still_running);

                if(still_running) {
                        /* wait for activity, timeout or "nothing" */
                        mc = curl_multi_poll(multi_handle, NULL, 0, 1000, &numfds);
                }

//              if(mc != CURLM_OK) {
//                      fprintf(stderr, "curl_multi_wait() failed, code %d.\n", mc);
//                      break;
//              }
        }

        curl_multi_remove_handle(multi_handle, http_handle);
        curl_easy_cleanup(http_handle);
        curl_multi_cleanup(multi_handle);
        curl_global_cleanup();

        return 0;
}

Answer 1

You need to move curl_multi_add_handle and curl_multi_remove_handle inside the while loop. Below is the extract from curl documentation https://curl.se/libcurl/c/libcurl-multi.html

When a single transfer is completed, the easy handle is still left added to the >multi stack. You need to first remove the easy handle with curl_multi_remove_handle >and then close it with curl_easy_cleanup, or possibly set new options to it and add >it again with curl_multi_add_handle to start another transfer.