Python bs4 不从元素返回文本

Question

I am trying to scrape reverb.com to get the names of different instruments. I have found the element that holds the instrument name text, but for some reason the tags return blank. I will provide my code below. Any ideas as to why this might be happening?

import requests
from bs4 import BeautifulSoup as Soup

url = 'https://reverb.com/marketplace?query=jackson+guitars'

response = requests.get(url).text
soup = Soup(url, 'html.parser')
for item in soup.find_all('h4', class_='grid-card__title'):
    print(item.text)

If you go to the web site, there are many results for the search. The most I've been able to return is four results, and I usually get back an empty list. I checked, and they all appear to have that h4 with the same class. Any ideas as to why it is not returning all results? Thanks for your time.

Answer 1

In line 6, you are using the requests library to get the content under the URL provided. In the next step, you should use the response of this request as input for Beautiful Soup, not the URL address. Just update this single line:

soup = Soup(response, 'html.parser')

Answer 2

I did some research and noticed that the site does not immediately load all the content.

In order for all product titles to be successfully extracted, you need to use scroll pagination technique to scroll the page to the end and wait until the data is loaded. To accomplish this task, I decided to use playwright .

The evaluate() method takes as a parameter the JavaScript code needed to scroll the page. After scrolling, you need to wait a while for the data to load:

page.evaluate('window.scrollTo(0, document.querySelector("body").scrollHeight)')
time.sleep(3)

We pass the page with the loaded data to the soup object, extract the titles of the products and close the browser:

soup = BeautifulSoup(page.content(), 'lxml')

for item in soup.select('.grid-card__title'):
    print(item.text)

page.close()

Full code:

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
import time, lxml


def run(playwright):
    URL = 'https://reverb.com/marketplace?query=jackson+guitars'

    page = playwright.chromium.launch(headless=True).new_page()
    page.goto(URL)

    page.evaluate('window.scrollTo(0, document.querySelector("body").scrollHeight)')
    time.sleep(3)

    soup = BeautifulSoup(page.content(), 'lxml')

    for item in soup.select('.grid-card__title'):
        print(item.text)

    page.close()


with sync_playwright() as playwright:
    run(playwright)

Output:

JACKSON Jackson Pro Series Rhoads RR24 Maul Crackle (S/N:CYJ21000010) (11/03)
Final Reduction!  2005 Jackson USA Soloist SL-1 in Gloss Black Finish! OHSC
Jackson X Series SL3X DX Soloist Frost Byte Crackle
Jackson JS Series JS32T Jackson Trem Black/White
Jackson Jackson RRX24  Black with Yellow Bevels
... ohter results