How to create a local proxy to handle invalid headers in http response

Question

I need to do a simple GET request to a website and parse the response.

I was planning to use the following simple code

var httpClient =_httpClientFactory.CreateClient();
var initialRequest = await httpClient.GetAsync(_config.WebsiteUrl, cancellationToken);

However, the site was poorly made and sends an invalid header in its response, namely "X=Frame-Options", resulting in the following exception.

System.Net.Http.HttpRequestException: Received an invalid header name: 'X=Frame-Options'.

Unfortunately, I do not control the site and it is very unlikely that this issue will be fixed anytime soon. So a workaround is needed.

My idea is to create a local "Man in the middle", so to speak, that would intercept the response sent from the site before reaching my code, remove the offending header, then pass the response on.

I am unfamiliar with networking in C# and I was wondering if there are any good libraries or existing examples for this use case.

I have tried using the Flurl library, but unfortunately, it uses httpclient in its implementation so the exception is still thrown.

Or am I missing something obvious and there is some way to disable the header validation that throws the above exception?

Answer 1

You must set useUnsafeHeaderParsing to true in the web.config file if you are using the web platform or in the desktop of the App.confing file To allow unsecured requests to be received

<configuration>
  //..............................
  <system.net>
    <settings>
      <httpWebRequest useUnsafeHeaderParsing="true"/>
    </settings>
  </system.net>
  //..............................
</configuration>

or in code behind add following method

public static bool SetAllowUnsafeHeaderParsing(bool value)
{
    //Get the assembly that contains the internal class
    Assembly aNetAssembly = Assembly.GetAssembly(typeof(System.Net.Configuration.SettingsSection));
    if (aNetAssembly != null)
    {
        //Use the assembly in order to get the internal type for the internal class
        Type aSettingsType = aNetAssembly.GetType("System.Net.Configuration.SettingsSectionInternal");
        if (aSettingsType != null)
        {
           //Use the internal static property to get an instance of the internal settings class.
           //If the static instance isn't created allready the property will create it for us.
           object anInstance = aSettingsType.InvokeMember("Section", BindingFlags.Static | BindingFlags.GetProperty | BindingFlags.NonPublic, null, null, new object[] { });

           if (anInstance != null)
           {
              //Locate the private bool field that tells the framework is unsafe header parsing should be allowed or not
             FieldInfo aUseUnsafeHeaderParsing = aSettingsType.GetField("useUnsafeHeaderParsing", BindingFlags.NonPublic | BindingFlags.Instance);
             if (aUseUnsafeHeaderParsing != null)
             {
                aUseUnsafeHeaderParsing.SetValue(anInstance, value);
                return true;
             }
           }
        }
    }
    return false;
}

now use

SetAllowUnsafeHeaderParsing(true);
var httpClient =_httpClientFactory.CreateClient();
var initialRequest = await httpClient.GetAsync(_config.WebsiteUrl, cancellationToken);
var response = await initialRequest.Content.ReadAsStringAsync();