使用Selenium和PhantomJS抓取表格

Question

I am trying to scrape the following table :

My code is working when I am using the chrome web driver but when using PhantomJS driver the output doesn't seem to get the numbers, it only gets the text.

My Python code is this :

    from selenium import webdriver

path_to_chromedriver = '/Users/amr_f/Desktop/chromedriver' # change path as needed
browser = webdriver.PhantomJS('/home/ubuntu/phantomjs-2.1.1-linux-x86_64/bin/phantomjs')
url = 'http://www.cibeg.com/English/Pages/default.aspx'
browser.get(url)

browser.find_element_by_xpath('//*[@id="sliderHome"]/div[2]/div/ul/li[3]/a').click()

data = []

for tr in browser.find_elements_by_xpath('//*[@id="divCurrTableContainer"]/table'):
    tds = tr.find_elements_by_tag_name('td')
    if tds: 

        data.append([td.text for td in tds])
print(data)

Answer 1

By adding, browser.set_window_size(1124, 850) , to set the window size for the PhantomJS driver I was able to retrieve the table's data from the page.

This happens, if I'm recalling this correctly, because certain javascript libraries use the window's size "on load". Not having the window size parameter can cause the routine to not correctly load all the elements on the page.

from selenium import webdriver


browser = webdriver.PhantomJS('/home/ubuntu/phantomjs-2.1.1-linux-x86_64/bin/phantomjs')
browser.set_window_size(1124, 850)
url = 'http://www.cibeg.com/English/Pages/default.aspx'
browser.get(url)    
browser.find_element_by_xpath('//*[@id="sliderHome"]/div[2]/div/ul/li[3]/a').click()    
data = []

for tr in browser.find_elements_by_xpath('//*[@id="divCurrTableContainer"]/table'):
    tds = tr.find_elements_by_tag_name('td')
    if tds:     
        data.append([td.text for td in tds])

print(data)

After I added the window size I was able to retrieve:

[['USD', '16.26', '16.75', 'EUR', '17.6696', '18.3563', 'GBP', '20.0895', '20.8621', 'CHF', '16.4571', '17.0536', 'SAR', '4.3297', '4.4663', 'KWD', '53.5202', '55.3353']]