HTTP 403 Forbidden coming when using HttpClient authentication mechanism for Sharepoint using NTLM

Question

HTTP/1.1 403 FORBIDDEN Response content length: 729 Result: soap:ServerException of type 'Microsoft.SharePoint.SoapServer.SoapServerException' was thrown.The security validation for this page is invalid. Click Back in your Web browser, refresh the page, and try your operation again.0x8102006d

My concern is how to get authenticated by proxy and also at the same time, to NTLM configured sharepoint in same session? Please help me.

Thanks in advance.

AuthCache authCache = new BasicAuthCache();
HttpHost targetHost = new HttpHost(<sharepointserverhostname>);
NTLMSchemeFactory f = new NTLMSchemeFactory();
HttpContext ctx = new BasicHttpContext();
AuthScheme ns = f.create(ctx);
authCache.put(targetHost, ns);

HttpClientContext localContext = HttpClientContext.create();
localContext.setAuthCache(authCache);

CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
    new AuthScope(<proxyserverip>, 8080), new UsernamePasswordCredentials  ("testdomain\phanigandeed", "Jul@2014"));
credsProvider.setCredentials(
            AuthScope.ANY,  new NTCredentials("phanigandeed", "Jul@2014", "", "testdomain"));

CloseableHttpClient httpclient = HttpClients.custom().setDefaultCredentialsProvider(credsProvider).build();

String listName = "PhaniList";
String description = "TestDescription";
String templateID = "101";
// for safety reasons, I had to remove the actual server details.
String endpointURL = <serviceurl>;  
String result = "Failed";
String username = "phanigandeed";
String psWord = "Jul@2014";
String domainName = "vsnl";
String XML_DATA = new String("<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\"><soap:Body><AddList xmlns=\"http://schemas.microsoft.com/sharepoint/soap/\"><listName>"
                    + listName
                    + "</listName><description>"
                    + description
                    + "</description><templateID>"
                    + templateID
                    + "</templateID></AddList></soap:Body></soap:Envelope>");

HttpPost httpPost = new HttpPost(endpointURL);
httpPost.setHeader(new BasicHeader("Content-Type",
            "text/xml;charset=UTF-8"));
try {
StringEntity s = new StringEntity(XML_DATA, "UTF-8");
httpPost.setEntity(s);

System.out.println("executing request" + httpPost.getRequestLine());

HttpResponse response = httpclient.execute(httpPost, localContext);
HttpEntity entity = response.getEntity();

System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
    System.out.println("Response content length: "
                    + entity.getContentLength());
    result = EntityUtils.toString(entity);
    System.out.println("Result: " + result);
    entity.consumeContent();
}
System.out.println("result: " + result);

return;
} catch (Exception e) {
e.printStackTrace();
System.out.println("Sharepoint Create Library failed :"
                + e.getMessage());

return;
}    

Answer 1

You need connect to SharePoint and retrieve a digest for authenticating any further requests, that is why it is throwing an exception about your security validation being invalid.

        String digest;
        try
        {
            string url = "https://Your.SharePoint.Site";
            HttpClient client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });
            client.BaseAddress = new System.Uri(url);
            string cmd = "_api/contextinfo";
            client.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
            client.DefaultRequestHeaders.Add("ContentType", "application/json");
            client.DefaultRequestHeaders.Add("ContentLength", "0");
            StringContent httpContent = new StringContent("");
            var response = client.PostAsync(cmd, httpContent).Result;
            if (response.IsSuccessStatusCode)
            {
                string content = response.Content.ReadAsStringAsync().Result;
                JsonObject val = JsonValue.Parse(content).GetObject();
                JsonObject d = val.GetNamedObject("d");
                JsonObject wi = d.GetNamedObject("GetContextWebInformation");
                digest = wi.GetNamedString("FormDigestValue");
            }
        }
        catch
        {
            MessageDialog dialog = new MessageDialog("Authentication to server failed.  Please try again.");
            dialog.ShowAsync();
        }

Later you will need to add the digest to the header of your next request to pass along the authentication which would look something like this. This is C# and sorry I don't know the Java specific syntax but you need the Digest to work with Web Services in all platforms:

client.DefaultRequestHeaders.Add("X-RequestDigest", digest);

Answer 2

I have had issues with HTTPClient working with Sharepoint/IIS and NTLM authentication. I was able to get past these issue when I switched to Java HTTPURLConnection. Please see the following posts for reference

HTTP Client and NTLM issue

HTTP Call to Sharepoint fails

Hope this helps you.