如何使用 libcurl 获取网页的标题

Question

How can I get the title of web page with curl? I wish to pass http or https url and get title of that page. I figured out that curl_easy_perform(curl) prints the html to terminal but I can't figure out how I can parse the html.

Answer 1

libcurl is not a HTML parsing library, it's focus is on the transport, ie getting you the bits. You need to either interpret them yourself, or turn to other libraries.

In your case you need to look for the <title> element and extract that element's text.

It's a bit too large to paste here, but this example from libcurl shows how to save content in memory in C. It uses the curl_easy_setopt() function to register a CURLOPT_WRITEFUNCTION callback which receives all the data.

Note that the libcurl example uses an "exact-fitting" dynamic string, ie it calls realloc() every time it gets more data. This is generally not the best approach but it's of course simple to implement and understand and might make sense in an example.

Answer 2

libcurl doesn't parse html for you. You need to use other libraries for that or write your own parser.

Have a look at HTML tidy. The Lib curl page has an example .

If you want just a title you may try a simple solution using std::string search or regular expressions.

Answer 3

#include <regex>
#include <unordered_map>
#include <string>
#include <iostream>
#include <curl/curl.h>

//Convert curl out to string
size_t curl_to_string(char* ptr, size_t size, size_t nmemb, void* data)
{
    std::string* str = (std::string*)data;
    int x;

    for (x = 0; x < size * nmemb; ++x)
    {
        (*str) += ptr[x];
    }

    return size * nmemb;
}

std::string curlGetHtmlSource(std::string& link)
{
    CURL* curl;
    CURLcode res;
    std::string html_txt;
    curl = curl_easy_init();
    if (curl) {
        curl_easy_setopt(curl, CURLOPT_URL, &link[0]);
        curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_to_string);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, &html_txt);

        /* Perform the request, res will get the return code */
        res = curl_easy_perform(curl);

        /* Check for errors */
        if (res != CURLE_OK)
        {
            fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
                curl_easy_cleanup(curl);
                throw std::runtime_error("Can't get html source");
        }

        /* always cleanup */
        curl_easy_cleanup(curl);
    }
    
    return html_txt;
}

//Pars regex from text
std::string stringRegex(const std::string& in_string , const std::string& regx)
{
    const std::regex pattern{ regx };

    for (
        std::sregex_iterator it(in_string.begin(), in_string.end(), pattern);
        it != std::sregex_iterator{};
        ++it
        )                                                               
    {
        return (*it)[1];
    }
    return {};
}

//Replace html entities
std::string entityParser(std::string text) {
    std::unordered_map<std::string, std::string> convert({
        {"&quot;", "\""},
        {"&apos;", "'"},
        {"&amp;", "&"},
        {"&gt;", ">"},
        {"&lt;", "<"},
        {"&frasl;", "/"}
        });
    std::string res = "";
    for (int i = 0; i < text.size(); ++i) 
    {
        bool flag = false;
        for (auto it = begin(convert); it != end(convert); ++it) 
        {
            std::string key = it->first;
            std::string value = it->second;
            if (i + key.size() - 1 < text.size()) 
            {
                if (text.substr(i, key.size()) == key) 
                {
                    res += value;
                    i += static_cast<int>(key.size() - 1);
                    flag = true;
                    break;
                }
            }
        }
        if (!flag) {
            res += text[i];
        }
    }
    return res;
}

std::string getTitle(std::string& link)
{
    std::string title = curlGetHtmlSource(link);
    title = stringRegex(title, R"(<title>([^<]*)<)");
    title = entityParser(title);
    return title ;
}

int main()
{
    std::string link = "https://example.com";
    getTitle(link);
}

Sources:

curl_to_string - https://stackoverflow.com/a/5525631/17061201

curlGetHtmlSource- https://curl.se/libcurl/c/simple.html

entityParser- https://helloacm.com/a-simple-html-entity-parser-in-c/