HTML解析器无法在网页上找到表格元素

Question

I'm trying to get to this element: //*[@id="table-matches"]/table on this page: http://www.oddsportal.com/matches/soccer/20140221/

I want to get the table that contains matches. Table starts under Kick off time tab. The element I'm looking for is 'table class=" table-main"' and it is inside the element 'div id="table-matches" style="display: block;"'

I tried getting this document with HtmlAgilityPack in C# and I can find 'div' element, but it says that it doesn't have any child nodes (there should be a table child node). If I try to get the table, the result is null. Here is the code:

var webGet = new HtmlWeb();
var document = webGet.Load("http://www.oddsportal.com/matches/soccer/20140221/");
var div = document.DocumentNode.SelectNodes("//div[@id='table-matches']");
var table = document.DocumentNode.SelectNodes("//*[@id='table-matches']/table");
var table2 = document.DocumentNode.SelectNodes("//table");

So, div variable contains the div element (but it has no child nodes), table variable is null, even table2 variable contains 4 elements, but none of them are desired table.

I figured there is a problem with HtmlAgilityPack and tried to get the whole web page with Python. So I got the whole HTML document in a text file and searched the text file and I can find div element but it is empty. There is no table element inside. Why is that? Why can I see table element in chrome or internet explorer, but when I download html there is no such element?

Here is the python code:

url = urllib.urlopen("http://www.oddsportal.com/matches/")
document = url.read()
htmlOddsPortal = open("htmlOddsPortal.txt", "w")
htmlOddsPortal.write(document)

Here is the element in the final text document:

<div id="table-matches"></div>                    <!--  END PAGE BODY -->

Answer 1

Table is loaded with JavaScript (probably with AJAX) so you won't get it with webGet.Load(). You only get HTML that server returns in response.

You can check this if you (in Chrome) open Console (F12), click on Settings and check Disable JavaScript, then refresh page. You will see blank content.

I had same problem, but I worked in java, and I have used HTMLUnit to solve this. Probably there is similar tool for C#, or you can check if HtmlAgilityPack is able to do asynchronous call or something like WebBrowser component.