来自WebClient的错误：“远程服务器返回错误：（503）服务器不可用”

Question

using (var wc = new WebClient())
{
    Uri urls = new Uri(url);
    wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36";
    dokuman.Load(wc.OpenRead(urls), Encoding.UTF8);
}

dokuman.Load(wc.OpenRead(urls), Encoding.UTF8);

The remote server returned an error: (503) Server Unavailable.

Sometimes I get this error. How can we solve this?

Answer 1

I use this utility function it supports proxies too

 public static string GetPageHtml(string link, System.Net.WebProxy proxy = null)
        {
            System.Net.WebClient client = new System.Net.WebClient() { Encoding = Encoding.UTF8 };
            client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
            if (proxy != null)
            {
                client.Proxy = proxy;
            }

            using (client)
            {
                try
                {
                    return client.DownloadString(link);
                }
                catch (Exception ex)
                {
                    return null;
                }
            }

        }

Try it, if it doesn't work then you probably got blocked by google because you were making requests fast, to avoid getting blocked you should use web proxies or make reasonable pauses between your requests.

Example

var Html = GetPageHtml("https://www.google.com.tr/search?sclient=psy-ab&site=&source=hp&q=cars+&btnG=Ara");