简体   繁体   中英

Can't get all titles from a list with Python WebScraping

I'm practicing web scraping with Python atm and I found a problem, I wanted to scrap one website that has a list of anime that I watched before but when I try to scrap it (via requests or selenium) it only gets around 30 of 110 anime names from the page. Here is my code with selenium:

from selenium import webdriver
from bs4 import BeautifulSoup

browser = webdriver.Firefox()
browser.get("https://anilist.co/user/Agusmaris/animelist/Completed")
data = BeautifulSoup(browser.page_source, 'lxml')
for title in data.find_all(class_="title"):
    print(title.getText())

And when I run it, the page source only shows up until an anime called 'Golden time' when there are like 70 or more left that are in the page.

Thanks

Edit: Code that works now thanks to 'supputuri':

from selenium import webdriver
from bs4 import BeautifulSoup
import time

driver = webdriver.Firefox()
driver.get("https://anilist.co/user/Agusmaris/animelist/Completed")
time.sleep(3)
footer = driver.find_element_by_css_selector("div.footer")
preY = 0
print(str(footer))
while footer.rect['y'] != preY:
    preY = footer.rect['y']
    footer.location_once_scrolled_into_view
    print('loading')
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')
for title in soup.find_all(class_="title"):
    print(title.getText())
driver.close()
driver.quit()
ret = input()

So, this is the jist of what I get when I load the page source:

AniListwindow.al_token = 'E1lPa1kzYco5hbdwT3GAMg3OG0rj47Gy5kF0PUmH';Sorry, AniList requires Javascript.
Please enable Javascript or http://outdatedbrowser.com>upgrade to a modern web browser.Sorry, AniList requires a modern browser.
Please http://outdatedbrowser.com>upgrade to a newer web browser.

Since I know damn well that Javascript is enabled and my Chrome version is fully up to date, and the URL listed takes one to a nonsecure website to "download" a new version of your browser, I think this is a spam site. Not sure if you were aware of that when posting so I won't flag as such, but I wanted you and others who come across this to be aware.

Here is the solution. Make sure to add import time

driver.get("https://anilist.co/user/Agusmaris/animelist/Completed")
time.sleep(3)
footer =driver.find_element_by_css_selector("div.footer")
preY =0
while footer.rect['y']!=preY:
    preY = footer.rect['y']
    footer.location_once_scrolled_into_view
    time.sleep(1)
print(str(driver.page_source))

This will iterate until all the anime is loaded and then gets the page source. Let us know if this was helpful.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM