HttpWebRequest For Atom URL returns HTML Content

Question

I'm trying to load an an Atom feed into a string and I'm using the following method:

private string GetXml(string urlString)
        {
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11;
            StringBuilder sb = new StringBuilder();
            // Create a new HttpWebRequest object.Make sure that 
            // a default proxy is set if you are behind a firewall.
            HttpWebRequest myHttpWebRequest1 =
              (HttpWebRequest)WebRequest.Create(urlString);

            myHttpWebRequest1.KeepAlive = false;
            myHttpWebRequest1.ContentType = "text/xml";
            // Assign the response object of HttpWebRequest to a HttpWebResponse variable.
            HttpWebResponse myHttpWebResponse1 =
              (HttpWebResponse)myHttpWebRequest1.GetResponse();

            Debug.WriteLine("\nThe HTTP request Headers for the first request are: \n{0}", myHttpWebRequest1.Headers);

            Stream streamResponse = myHttpWebResponse1.GetResponseStream();
            StreamReader streamRead = new StreamReader(streamResponse);
            Char[] readBuff = new Char[256];
            int count = streamRead.Read(readBuff, 0, 256);
            Debug.WriteLine("The contents of the response are.......\n");
            while (count > 0)
            {
                String outputData = new String(readBuff, 0, count);
                sb.Append(outputData);
                count = streamRead.Read(readBuff, 0, 256);
            }
            // Close the Stream object.
            streamResponse.Close();
            streamRead.Close();
            return sb.ToString();
        }

This works for most feeds, but http://tipsforbbq.com/RSS , which renders as an atom feed in my browser, returns an HTML page (the one found at http://tipsforbbq.com/ ). Does anyone know why this might be happening?

Answer 1

I had a similar problem with HttpWebRequest and I resolved it using a well known User Agent.

myHttpWebRequest1.UserAgent= @"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1";

I hope this can help you.