WebClient generates (401) Unauthorized error

Question

I have the following code running in a windows service:

WebClient webClient = new WebClient();
webClient.Credentials = new NetworkCredential("me", "12345", "evilcorp.com");
webClient.DownloadFile(downloadUrl, filePath);

Each time, I get the following exception

{"The remote server returned an error: (401) Unauthorized."}

With the following inner exception:

{"The function requested is not supported"}

I know for sure the credentials are valid, in fact, if I go to downloadUrl in my web browser and put in my credentials as evilcorp.com\\me with password 12345, it downloads fine.

What is weird though is that if I specify my credentials as me@evilcorp.com with 12345, it appears to fail.

Is there a way to format credentials?

Answer 1

webClient.UseDefaultCredentials = true; resolved my issue.

Answer 2

Apparently the OS you are running on matters, as the default encryption has changed between OSes. This blog has more details: http://ferozedaud.blogspot.com/2009/10/ntlm-auth-fails-with.html

This has apparently also been discussed on stackoverflow here: 407 Authentication required - no challenge sent

I would suggest read the blog first as the distilled knowledge is there.

Answer 3

According to the msdn docs the exception could be because the method has been called simultaneously on multiple threads. The DownloadFile method also requires a completely qualified URL such as http://evilcorp.com/ .

Answer 4

For me, 'webClient.UseDefaultCredentials = true;' solves it only on local, not in the web app on the server connecting to another server. I couldn't add the needed credential into Windows as a user, but I found later some programming way - I won't test it as I already made own solution. And also I don't want to mangle with the web server's registry , even if I have the needed admin rights. All these problems are because of the Windows internal handling of the NTLM authentication ("Windows Domain") and all of libraries and frameworks built over that (eg .NET).

So the solution for me was quite simple in idea - create a proxy app in a multiplatform technology with a multiplatform NTLM library where the NTLM communication is created by hand according to the public specs, not by running the built-in code in Windows. I myself chose Node.js and the httpntlm library , because it's about only one single source file with few lines and calling it from .NET as a program returning the downloaded file (also I prefer transferring it through the standard output instead of creating a temporary file).

Node.js program as a proxy to download a file behind the NTLM authentication:

var httpntlm = require('httpntlm');         // https://github.com/SamDecrock/node-http-ntlm
//var fs = require('fs');

var login = 'User';
var password = 'Password';
var domain = 'Domain';

var file = process.argv.slice(2);           // file to download as a parameter

httpntlm.get({
    url: 'https://server/folder/proxypage.aspx?filename=' + file,
    username: login,
    password: password,
    workstation: '',
    domain: domain,
    binary: true                            // don't forget for binary files
}, function (err, res/*ponse*/) {
    if (err) { 
        console.log(err);
    } else {
        if (res.headers.location) {         // in my case, the server redirects to a similar URL,
            httpntlm.get({                  // now containing the session ID
                url: 'https://server' + res.headers.location,
                username: login,
                password: password,
                workstation: '',
                domain: domain,
                binary: true                // don't forget for binary files
            }, function (err, res) {
                if (err) { 
                    console.log(err);
                } else {
                      //console.log(res.headers);
                      /*fs.writeFile("434980.png", res.body, function (err) {  // test write
                          if (err)                                             // to binary file
                              return console.log("Error writing file");
                          console.log("434980.png saved");
                      });*/
                      console.log(res.body.toString('base64'));  // didn't find a way to output
                }                                                // binary file, toString('binary')
            });                                                  // is not enough (docs say it's
                                                                 // just 'latin1')...
        } else {           // if there's no redirect
            //console.log(res.headers);                          // ...so I output base64 and
            console.log(res.body.toString('base64'));            // convert it back in the caller 
        }                                                        // code
    }
});

.NET caller code (the web app downloading files from a web app on another server)

public static string ReadAllText(string path)
{
    if (path.StartsWith("http"))
        return System.Text.Encoding.Default.GetString(ReadAllBytes(path));
    else
        return System.IO.File.ReadAllText(path);
}

public static byte[] ReadAllBytes(string path)
{
    if (path.StartsWith("http"))
    {
        ProcessStartInfo psi = new ProcessStartInfo();
        psi.FileName = "node.exe";                     // Node.js installs into the PATH
        psi.Arguments = "MyProxyDownladProgram.js " + 
            path.Replace("the base URL before the file name", "");
        psi.WorkingDirectory = "C:\\Folder\\With My\\Proxy Download Program";
        psi.UseShellExecute = false;
        psi.CreateNoWindow = true;
        psi.RedirectStandardInput = true;
        psi.RedirectStandardOutput = true;
        psi.RedirectStandardError = true;
        Process p = Process.Start(psi);
        byte[] output;
        try
        {
            byte[] buffer = new byte[65536];
            using (var ms = new MemoryStream())
            {
                while (true)
                {
                    int read = p.StandardOutput.BaseStream.Read(buffer, 0, buffer.Length);
                    if (read <= 0)
                        break;
                    ms.Write(buffer, 0, read);
                }
                output = ms.ToArray();
            }
            p.StandardOutput.Close();
            p.WaitForExit(60 * 60 * 1000);             // wait up to 60 minutes 
            if (p.ExitCode != 0)
                throw new Exception("Exit code: " + p.ExitCode);
        }
        finally
        {
            p.Close();
            p.Dispose();
        }
        // convert the outputted base64-encoded string to binary data
        return System.Convert.FromBase64String(System.Text.Encoding.Default.GetString(output));
    }
    else
    {
        return System.IO.File.ReadAllBytes(path);
    }
}

Answer 5

Hmm. Lots of answers, but I wonder if answering your last question would have solved everything. "me" is not an authorization type (unless your server has added support for it, of course!). You probably want "Basic".

Also keep in mind that some webservices require you to send the authorization header on the initial request, and this won't do that. Rather it responds with it after getting an authorization required response from the server. If you need this, you need to create your own Authorization header.

String basicToken = Base64Encoding.EncodeStringToBase64(String.Format("{0}:{1}", clientId.Trim(), clientSecret.Trim()));

webClient.Headers.Add("Authorization", String.Format("Basic {0}", basicToken));

And of course as people have pointed out, setting UseDefaultCredentials to true works if you are using IIS (or other windows security aware http server) in a windows environment.

Answer 6

While writing a URL, put the '@' in front of url string.

For example:

var url = @"http://evilcorp.com";
WebClient webClient = new WebClient();
webClient.Proxy = null;
webClient.DownloadFile(new Uri(url), filePath);

This will solve your problem.

The answer given by P.Brian.Mackey is also correct.