MSXML instead of IE in VBA produces runtime error 91 “Object variable or With block variable not set”

Question

I'm examining almost 10,000 web pages and returning values to an Exel spreadsheet. Using an IE object produces inconsistent results and I want to use MSXML instead so I've produced two functions that return the web page, one using IE and another using MSXML. They should return the same result. Here is the IE function:

Function getIEPage(link) As HTMLDocument
Dim objIE As Object
Set objIE = CreateObject("InternetExplorer.Application")
With objIE
    .Top = 0
    .Left = 0
    .Width = 800
    .Height = 600
    .Visible = False
End With
objIE.navigate (link)
While objIE.readyState <> 4
    DoEvents
Wend
Set getIEPage = objIE.document
End Function

Here is the XML function:

Function getXMLPage(link) As MSHTML.HTMLDocument
Dim ie As MSXML2.XMLHTTP60
Set ie = New MSXML2.XMLHTTP60
ie.Open "GET", link, False
ie.send
While ie.readyState <> 4
    DoEvents
Wend
Dim HTMLDoc As MSHTML.HTMLDocument
Set HTMLDoc = New MSHTML.HTMLDocument
HTMLDoc.body.innerHTML = ie.responseText
Set getXMLPage = HTMLDoc
End Function

At the next step, I want to extract all the "a" tags, like this:

Dim webPage As HTMLDocument
Dim allLinks1 As Variant
Set webPage = getIEPage("https://www.stackoverflow.com/")
Set allLinks1 = webPage.getElementsByTagName("A")

or like this:

Dim webPage As HTMLDocument
Dim allLinks2 As Variant
Set webPage = getXMLPage("https://www.stackoverflow.com/")
Set allLinks2 = webPage.getElementsByTagName("A")

But these do not produce the same result. The IE function executes but the XML produces a runtime error 91 "Object variable or With block variable not set"

Answer 1

You are dealing with HTML. You can therefore do the following to get all the a tags:

Option Explicit
Public Sub GetTagList()
    Dim sResponse As String, tagList As Object
    With CreateObject("MSXML2.XMLHTTP")
        .Open "GET", "https://stackoverflow.com/", False
        .send
        sResponse = StrConv(.responseBody, vbUnicode)
    End With
    With CreateObject("htmlFile")
        .Write sResponse
        Set tagList = .getElementsByTagName("a")
        Debug.Print tagList.Length
    End With
End Sub