简体   繁体   English

如何在不下载所有页面源的情况下获取网页标题

[英]How to get webpage title without downloading all the page source

I'm looking for a method that will allow me to get the title of a webpage and store it as a string. 我正在寻找一种方法,可以让我获得网页的标题并将其存储为字符串。

However all the solutions I have found so far involve downloading the source code for the page, which isn't really practical for a large number of webpages. 但是到目前为止我找到的所有解决方案都涉及下载页面的源代码,这对于大量网页来说并不实用。

The only way I could see would be to limit the length of the string or it only downloads either a set number of chars or stops once it reaches the tag, however this obviously will still be quite large? 我能看到的唯一方法是限制字符串的长度,或者一旦到达标签,它只下载一定数量的字符或停止,但这显然仍然会很大?

Thanks 谢谢

As the <title> tag is in the HTML itself, there will be no way to not download the file to find "just the title". 由于<title>标签位于HTML本身,因此无法下载文件以找到“只是标题”。 You should be able download a portion of the file until you've read in the <title> tag, or the </head> tag and then stop, but you'll still need to download (at least a portion of) the file. 你应该可以下载文件的一部分,直到你读入<title>标签或</head>标签然后停止,但你仍然需要下载(至少一部分)文件。

This can be accomplished with HttpWebRequest / HttpWebResponse and reading in data from the response stream until we've either read in a <title></title> block, or the </head> tag. 这可以通过HttpWebRequest / HttpWebResponse完成,并从响应流中读取数据,直到我们读入<title></title>块或</head>标记。 I added the </head> tag check because, in valid HTML, the title block must appear within the head block - so, with this check we will never parse the entire file in any case (unless there is no head block, of course). 我添加了</head>标签检查,因为在有效的HTML中,标题栏必须出现在头部块中 - 因此,通过此检查,我们将永远不会解析整个文件(当然,除非没有头部块,否则)。

The following should be able to accomplish this task: 以下应该能够完成这个任务:

string title = "";
try {
    HttpWebRequest request = (HttpWebRequest.Create(url) as HttpWebRequest);
    HttpWebResponse response = (request.GetResponse() as HttpWebResponse);

    using (Stream stream = response.GetResponseStream()) {
        // compiled regex to check for <title></title> block
        Regex titleCheck = new Regex(@"<title>\s*(.+?)\s*</title>", RegexOptions.Compiled | RegexOptions.IgnoreCase);
        int bytesToRead = 8092;
        byte[] buffer = new byte[bytesToRead];
        string contents = "";
        int length = 0;
        while ((length = stream.Read(buffer, 0, bytesToRead)) > 0) {
            // convert the byte-array to a string and add it to the rest of the
            // contents that have been downloaded so far
            contents += Encoding.UTF8.GetString(buffer, 0, length);

            Match m = titleCheck.Match(contents);
            if (m.Success) {
                // we found a <title></title> match =]
                title = m.Groups[1].Value.ToString();
                break;
            } else if (contents.Contains("</head>")) {
                // reached end of head-block; no title found =[
                break;
            }
        }
    }
} catch (Exception e) {
    Console.WriteLine(e);
}

UPDATE: Updated the original source-example to use a compiled Regex and a using statement for the Stream for better efficiency and maintainability. 更新:更新了原始源代码示例,以便为Stream使用已编译的Regexusing语句,以提高效率和可维护性。

A simpler way to handle this would be to download it, then split: 处理此问题的一种更简单的方法是下载它,然后拆分:

    using System;
    using System.Net.Http;

    private async void getSite(string url)
    {
        HttpClient hc = new HttpClient();
        HttpResponseMessage response = await hc.GetAsync(new Uri(url, UriKind.Absolute));
        string source = await response.Content.ReadAsStringAsync();

        //process the source here

    }

To process the source, you can use the method described here in the article on Getting Content From Between HTML Tags 要处理源,您可以使用“ 从HTML标记之间获取内容 ”一文中所述的方法

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM