使用Selenium提取数据并导出到csv文件

Question

I'm trying to scrape all data from this page: https://icostats.com . I need to export the content of each row to a row in a csv file. Also, I think there's got to be a prettier and more functional way of iterating though each row. So far, here's where I'm at:

    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.ui import WebDriverWait as wait

    def get_css_sel(selector):
        posts = browser.find_elements_by_css_selector(selector)
        for post in posts:
            print(post.text)

    browser = webdriver.Chrome(executable_path=r'C:\Scrapers\chromedriver.exe')
    browser.get("https://icostats.com")
    wait(browser, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#app > div > div.container-0-16 > div.table-0-20 > div.tbody-0-21 > div:nth-child(2) > div:nth-child(8)")))

    get_css_sel("#app > div > div.container-0-16 > div.table-0-20 > div.tableheader-0-50")              #fetch header of table
    get_css_sel("#app > div > div.container-0-16 > div.table-0-20 > div.tbody-0-21 > div:nth-child(1)") #fetch values from 2nd row
    get_css_sel("#app > div > div.container-0-16 > div.table-0-20 > div.tbody-0-21 > div:nth-child(2)") #fetch values from 3rd row[...]
    [...]
    get_css_sel("#app > div > div.container-0-16 > div.table-0-20 > div.tbody-0-21 > div:nth-child(28)") #[...]fetch values from last row

EDIT

As for the row iteration, I thought something like this might do it:

for row in rows(first, last):
    row += 1
    get_css_sel("#app > div > div.container-0-16 > div.table-0-20 > div.tbody-0-21 > div:nth-child(row)")

But I can't figure out how to express that into actual code.

Answer 1

You could get manually the css path of the elements (you can do it in Chrome) and then just format the string with:

get_css_sel("string({})".format(row))

For example: this image

Hope this helps!

Edit:

This is how you would iterate over it:

for row in range(first, last): #first is included but not last
    get_css_sel("#app > div > div.container-0-16 > div.table-0-20 > div.tbody-0-21 > div:nth-child({})".format(row)) #assuming path is well constructed
    row += 1

I think this will do.