SSIS 脚本：部分 Uri 上的 HTTPWebRequest 和 Webclient.Downloadfile 失败

Question

I am trying to download a file from an API, which works perfectly well in chrome using the following Url:

https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/namq_10_pc/.CP_EUR_HAB.NSA.B1GQ.?startPeriod=2017&format=SDMX_2.1_STRUCTURED

However, when I try to download this in ac# script in SSIS using either HTTPWebRequest or Webclient.Downloadfile it fails (404 response error).

After many hours scouring Stackoverflow, trying numerous approaches, I have found that both approaches do work if I remove part of the Url:

https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/namq_10_pc/?startPeriod=2017&format=SDMX_2.1_STRUCTURED

Why does the inclusion of ".CP_EUR_HAB.NSA.B1GQ." make it fail in c# but not in chrome?

The code I am using:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(sourceFilePath));
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (Stream file = File.Create(savePathAndName))
{
     CopyStream(stream, file);
}

public static void CopyStream(Stream input, Stream output)
{
    byte[] buffer = new byte[8 * 1024];
    int len;
    while ((len = input.Read(buffer, 0, buffer.Length)) > 0)
    {
         output.Write(buffer, 0, len);
    }
}

UPDATE Originally, SSIS was omitted, as I thought it would be irrelevant, however, after the comment saying that they could get it working, I tested what the Uri is returning, and it is removing the last dot before the question mark:

https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/namq_10_pc/.CP_EUR_HAB.NSA.B1GQ?startPeriod=2017&format=SDMX_2.1_STRUCTURED

There are many documented difficulties in getting these dot segments included: just a couple include:

Is there a way to keep dot segments in url using Uri class? and A url resource that is a dot (%2E)

The API returns a failure if that dot isn't there. And microsoft confirms this:

https://docs.microsoft.com/en-us/dotnet/api/system.uri?view=net-5.0

After some investigation, it seems that this might be to do with the framework, and updating it might be the answer, which might suggest why others could get this working, but not me in SSIS. The framework that SSIS 2017 is using is bound to 4.5. I'm thinking that I might be at a dead-end with this.

Answer 1

It was the trailing dot in the section ".CP_EUR_HAB.NSA.B1GQ." that was causing the issue. the Uri removes dot segments automatically, and it appears there is no way round this.

With it being used in SSIS 207, there is no way for me to change the target framework, which Microsoft now ties to the version of SSIS.

The solution was to instead of having the trailing "." which the API requires to specify the country wild card, I had to instead list each country code in the URL:

https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/namq_10_pc/.CP_EUR_HAB.NSA.B1GQ.UK+BE+BG+CZ+DK+DE+EE+IE+EL+ES+FR+HR+IT+CY+LV+LT+LU+HU+MT+NL+AT+PL+PT+RO+SI+SK+FI+SE+IS+NO+CH+ME+MK+AL+RS+TR+BA+XK?startPeriod=2017&format=SDMX_2.1_STRUCTURED