Web data scraping html table Excel VBA

Question

I have no problems getting data from a table on the webpage, but I bumped into two issues.

On the webpage, it only shows one page of table data, at the bottom I can click page 2, 3 and NEXT. Using Ctrl + U to see the html codes, I can see all information are loaded and neatly organised as tr rows.

I assume ie.document can store all elements, as what I can see using Ctrl + U in Chrome. And when I try to retrieve the tr innertext by using getelementByTagName("tr") , vba just skipped all tr elements which is supposed to be on other pages of the table. Pardon me if I don't explain well, and I couldn't paste the link and html codes here as it may contain confidential information. I wonder if someone ever bumped into similar issues.

ieDoc = ieApp.Document

For Each HTMLRow In ieDoc.getElementsByTagName("tr")

Debug.print HTMLRow.innertext

Answer 1

Since I cannot comment yet, I hope the following helps at least a little.

To obtain the entire set of elements, you have to define a variable like so:

Dim variableName as IHTMLElementCollection

To return the collection:

Set variableName = ie.document.getElementsByTagName("tr")

In order to use the above, make sure to add the Reference Microsoft Internet Controls under Tools > References.

I asked a similar question in the past which might give you a larger example of how this works Excel VBA getElementsByTagName