SSIS HTTP Connection Manager returns a cached result

Question

I created a SQL 2012 SSIS package which uses an HTTP connection, via HTTP Connection Manager, to request a file from a Web site. Because it's still in development, I run the package from Visual Studio. The package was working just fine until a week ago. Now, instead of a file, the requested page returns a page stating "Unable to find the output file."

Well. I copied the URL string from the connection manager and pasted it into an Edge browser. The download works just fine there; I get the file in the format I expect. It works with repeated requests. I've tried altering the URL (moving querystring variables around), but I still get the undesired result in SSIS.

Is there some sort of HTTP caching going on in SSIS within Visual Studio? If so, how can I clear it out and get an updated result from the page request? If not, why do I see the failed response repeatedly in Visual Studio, yet it works in the browser?

Answer 1

I always use a script task for web requests. Very simple to do so and always works.

System.Net.WebClient wc = new System.Net.WebClient();
wc.DownloadFile([url],[fileLocations]);

This is mostly because the usual response is typically Json or XML which is much easier and reliably processed with c# NuGet packages.

"Always works" needs a caveat. You need to have access to the URL.