Using HttpWebRequest and HttpWebResponse throws an error, however only on certain websites

Question

This function checks if a url is for a valid address, and works with http://www.google.com

public static bool CheckValidURL(string url)
{
    try
    {
        //Creating the HttpWebRequest
        HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
        //Setting the Request method HEAD, you can also use GET too.
        request.Method = "GET";
        //Getting the Web Response.
        HttpWebResponse response = request.GetResponse() as HttpWebResponse;
        //Returns TRUE if the Status code == 200
        bool isValid;
        if (response.StatusCode == HttpStatusCode.OK)
        {
            isValid = true;
        }
        else
        {
            isValid = false;
        }
        response.Close();
        return isValid;
    }
    catch
    {
        //Any exception will returns false.
        return false;
    }
}

This Code is called when a menu item is clicked and is meant to update a cards price. It is within a foreach loop where the variable c is of a custom class MagicCard .

string url = "";
string webCardName = c.name.ToLower().Replace(" ", "-").Replace(",", "").Replace("\'", "");
string webSetName = MagicCard.GetSetName(c).ToLower().Replace(" ", "-").Replace(",", "").Replace("\'", "");
url = string.Format("shop.tcgplayer.com/magic/{0}/{1}", webSetName, webCardName);
if (WebScraper.CheckValidURL(url) == false)
{
    MessageBox.Show("ERROR: URL = " + url);
    return;
}

an example of a final url is this Although this is a valid address, it is detected as not and any other card would produce similar url's which also do not work. Why is this the case?

Answer 1

Your site requires a User-Agent . I Added this line

request.UserAgent = "SO/1.0";

to your code and it worked....