Python 网页抓取“IndexError: list index out of range”

Question

from selenium import webdriver
import csv
import requests
from bs4 import BeautifulSoup

driver = webdriver.Chrome(executable_path="C:\\Users\\dylan\\Documents\\chromedriver.exe")
data_list=[]


site = requests.get('https://www.visitnh.gov/things-to-do/food-and-drink/restaurants')

if site.status_code is 200:
   content = BeautifulSoup(site.content, 'html.parser')
   Resultswrapper = content.find_all('div', attrs={'results-wrapper'})

for Results in Resultswrapper:
    print("Random phrase: ")
    #print(Results.select(class_='results-wrapper').prettify())
    BusinessName = Results.select('.item-title.ng-binding')[0].get_text()
    Address =   Results.select('.ng-binding')[0].get_text()
    PhoneNumber = Results.select('.ng-binding')[0].get_text()
    new_data = {"BusinessName": BusinessName, "Address": Address, "PhoneNumber": PhoneNumber}
    data_list.append(new_data)
    print("data: " + data_list)
    print("new data: " + new_data)

with open ('find.csv','w') as file:
        writer = csv.DictWriter(file, fieldnames = ["BusinessName", "Address", "PhoneNumber"], delimiter = ';')
        writer.writeheader()
        for row in data_list:
            writer.writerow(row)

I get an index out of range error when trying to make a simple web scraper and I'm trying to loop using a selector. I use this URL:https://www.visitnh.gov/things-to-do/food-and-drink/restaurants

Traceback (most recent call last):
File "c:/Users/dylan/Documents/Webscrape/web-s.py", line 19, in <module>
BusinessName = Results.select('p.item-title.ng-binding')[0].get_text()
IndexError: list index out of range

I tried change the result wrapper to something different in the HTMl but its not the same. I also tried messing around with the text in the select, but no use. Any ideas? Any help would be greatly appreciated.

Answer 1

Looking at the website, you can see that the element you are trying to select is not present when the document is first ready. The list of restaurants on this website is inserted into the page later, which is why your script cannot find it right away.

Maybe wait a bit before selecting with the built in time.sleep function.

Answer 2

The data is loaded from external URL via javascript. You can use requests / json module to simulate this request:

import json
import requests


url = 'https://www.visitnh.gov/BusinessListingService.asmx/GetUsers'
search_params = {"searchParams":{"ResultsPerPage":15,"BusinessType":"attraction","SubcategoryID":25}}
data = json.loads(requests.post(url, json=search_params).json()['d'])

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for result in data['Results']:
    print(result['BusinessName'])
    print(result['Address'])
    print(result['Phone'])
    print('-' * 80)

Prints:

NazBar & Grill
1086 Weirs Blvd, Laconia, NH 03246
(603) 366-4341
--------------------------------------------------------------------------------
Giant of Siam
5 E Hollis Street, Nashua, NH 03060
(603) 595-2222
--------------------------------------------------------------------------------
Hobbs Tavern & Brewing Company
2415 NH Route 16, Ossipee, NH 03890
(603) 539-2000
--------------------------------------------------------------------------------

...and so on. (Total 164 items.)