Excel VBA get data from web with msxml2.xmlhttp - how do I accept cookies automatically?

Question

need your expertise and help, as i've looked around and couldn't find a solution:

I am uploading information to Excel from a website using the msxml2.xmlhttp method (did it earlier via webquery but it gets stuck after a few iterations plus it is slower). My problem is that now on every iteration, I have a Windows Security warning popping up asking me to accept a cookie from the website. Note that the website doesn't require a login/password. I understood from an earlier post that the msxml2.xmlhttp method strips cookies for security reasons, but I get the same message even if I change the method to winhttp. I also changed the settings in IE to accept all cookies automatically from the website but it didn't help.

My question is, what code do I need to add in order to have the cookies be accepted automatically, as I am looping this code on bulk and can't have it hang waiting for me to accept the cookie manually. Your help will be very much appreciated!!! Below is the code snippet (which I actually found here on Stackoverflow).

    Set htm = CreateObject("htmlFile")
With CreateObject("msxml2.xmlhttp")
    .Open "GET", "http://finance.yahoo.com/q/ae?s=" & Ticker & "+Analyst+Estimates", False
    .send
    htm.body.innerHTML = .responseText
End With
Set elemCollection = htm.getElementsByTagName("td")
For Each itm In elemCollection
    If itm.className = "yfnc_tabledata1" Then
        ActiveCell = itm.innerText
        If ActiveCell.Column = 7 Then
            ActiveCell.Offset(1, -6).Select
        Else
            ActiveCell.Offset(0, 1).Select
        End If
    End If
Next

Answer 1

I had the same problem this week.

After google and trying some ideas, I added two MsgBox statements to my code.

objXMLDoc.Open "GET", strURL, False
objXMLDoc.send
MsgBox "After XMLDoc.send", vbOKOnly, "Test"
objHTMLDoc.body.innerHTML = objXMLDoc.responseText
MsgBox "After .innerHTML assignment", vbOKOnly, "Test"

I found pop-up security warning windows always appear after .innerHTML assignment, ie, the problem is nothing to do with XMLHttp. It is HTMLDocument, which causes the pop-ups.

I guess objHTMLDoc.body.innerHTML = objXMLDoc.responseText does not just do a simple value assignment. It must also trigged some action according to the contents of the webpage.

I checked the webpage and found some code like this:

YUI().use('node','event','event-mouseenter','substitute','oop','node-focusmanager','node','event','substitute','**cookie**','event-resize','node', 'event', 'querystring-stringify','node','event','node','event','event-custom','event-valuechange','classnamemanager','node', function(Y) {})

Then I changed my code as follows and the pop-up warning windows disappear.

objXMLDoc.Open "GET", strURL, False
objXMLDoc.send
objHTMLDoc.body.innerHTML = Replace(objXMLDoc.responseText, "cookie", "")

Hope this can be helpful if you still have the problem.