HttpClient GetAsync with a hash in URL

Question

.NET Core 2.2 console application on Windows.

I'm exploring how to use HttpClient GetAsync on a Stackoverflow share style URL eg: https://stackoverflow.com/a/29809054/26086 which returns a 302 redirect URL with a hash in it

static async Task Main()
{
    var client = new HttpClient();

    // 1. Doesn't work - has a hash in URL
    var url = "https://stackoverflow.com/questions/29808915/why-use-async-await-all-the-way-down/29809054#29809054";
    HttpResponseMessage rm = await client.GetAsync(url);
    Console.WriteLine($"Status code: {(int)rm.StatusCode}"); // 400 Bad Request

    // 2. Does work - no hash
    url = "https://stackoverflow.com/questions/29808915/why-use-async-await-all-the-way-down/29809054";
    rm = await client.GetAsync(url);
    Console.WriteLine($"Status code: {(int)rm.StatusCode}"); // 200 Okay

    // 3. Doesn't work as the 302 redirect goes to the first URL above with a hash
    url = "https://stackoverflow.com/a/29809054/26086";
    rm = await client.GetAsync(url);
    Console.WriteLine($"Status code: {(int)rm.StatusCode}"); // 400 Bad Request
}

I'm crawling my blog which has many SO short codes in it.

Update/Workaround With thanks to @rohancragg I found that turning off AutoRedirect then getting the URI from the returned header worked

// as some autoredirects fail due to #fragments in url, handle redirects manually
var handler = new HttpClientHandler { AllowAutoRedirect = false };
var client = new HttpClient(handler);

var url = "https://stackoverflow.com/a/29809054/26086";    
HttpResponseMessage rm = await client.GetAsync(url);

// gives the desired new URL which can then GetAsync
Uri u = rm.Headers.Location;

Answer 1

As @Damien_The_Unbeliever implies in a comment, you'll just need to strip off the hash and everything after it - all that does is tell the browser to jump to that anchor tag in the HTML page (see: https://w3schools.com/jsref/prop_anchor_hash.asp ).

You could also use the Uri class to parse the Uri and ignore any 'fragments': https://docs.microsoft.com/en-us/dotnet/api/system.uri.fragment

Because the share-style Urls are only ever going to return a 302 then I'd suggest capturing the Uri to which the 302 is referring and do as I suggest above and just get the path and ignore the fragment.

So you need to use some mechanism (which I'm just looking up!) to handle a 302 gracefully followed by option 2

Update: this looks relevant! How can I get System.Net.Http.HttpClient to not follow 302 redirects?

Update 2 Steve Guidi has a very important bit of advice in a comment here: https://stackoverflow.com/a/17758758/5351

In response to the advice that you need to use HttpResponseMessage.RequestMessage.RequestUri :

it is very important to add HttpCompletionOption.ResponseHeadersRead as the second parameter of the GetAsync() call

---

Disclaimer - I've not tried the above, this is just based on reading ;-)

Answer 2

Maybe you need to encode your URL before send the request using HttpUtility class, this way any special character will be escaped.

using System.Web;

var url = $"htpps://myurl.com/{HttpUtility.UrlEncode("#1234567")}";