列表索引超出范围 - 美丽的汤

Question

NEW TO PYTHON*** Below is my code I am using to pull a zip file from a website but I am getting the error, "list index out of range". I was given this code by someone else who wrote it but I had to change the URL and now I am getting the error. When I print(list_of_documents) it is blank.

Can someone help me with this? The url requires access so you won't be able to try to input this code directly. I am trying to understand how to use beautiful soup in this and how I can get the list to populate correctly.

import datetime
import requests
import csv
from zipfile import ZipFile as zf
import os
import pandas as pd
import time
from bs4 import BeautifulSoup
import pyodbc
import re

#set download location

downloads_folder = r"C:\Scripts\"


##### Creating outage dataframe

#Get list of download links

res = requests.get('https://www.ercot.com/mp/data-products/data-product-details?id=NP3-233-CD')

ercot_soup = BeautifulSoup(res.text, "lxml")

list_of_documents = ercot_soup.findAll('td', attrs={'class': 'labelOptional_ind'})
list_of_links = ercot_soup.select('a')'

##create the url for the download 

loc = str(list_of_links[0])[9:len(str(list_of_links[0]))-9]
link = 'http://www.ercot.com' + loc
link = link.replace('amp;','')

# Define file name and set download path

file_name = str(list_of_documents[0])[30:len(str(list_of_documents[0]))-5]
file_path = downloads_folder + '/' + file_name

Answer 1

You can't expect code tailored to scrape one website to work for a different link! You should always inspect and explore your target site, especially the parts you need to scrape, so you know the tag names [like td and a here] and identifying attributes [like name , id , class , etc.] of the elements you need to extract data from.

With this site, if you want the info from the reportTable , it gets generated after the page gets loaded with javascript, so it wouldn't show up in the request response. You could either try something like Selenium , or you could try retrieving the data from the source itself.

If you inspect the site and look at the network tab , you'll find a request (which is what actually retrieves the data for the table) that looks like this , and when you inspect the table's html , you'll find above it the scripts to generate the data.

In the suggested solution below, the getReqUrl scrapes your link to get the url for requesting the reports (and also the template of the url for downloading the documents).

def getReqUrl(scrapeUrl):
  res = requests.get(scrapeUrl)
  ercot_soup = BeautifulSoup(res.text, "html.parser")

  script = [l.split('"') for l in [
      s for s in ercot_soup.select('script')
      if 'reportListUrl' in  s.text 
      and 'reportTypeID' in s.text
  ][0].text.split('\n') if l.count('"') == 2]

  rtID = [l[1] for l in script if 'reportTypeID' in l[0]][0]
  rlUrl = [l[1] for l in script if 'reportListUrl' in l[0]][0]
  rdUrl = [l[1] for l in script if 'reportDownloadUrl' in l[0]][0]

  return f'{rlUrl}{rtID}&_={int(time.time())}', rdUrl

(I couldn't figure out how to scrape the last query parameter [the &_=... part] from the site exactly, but {int(time.time())}} seems to get close enough - the results are the same even then and even when that last bit is omitted entirely; so it's totally optional.)

The url returned can be used to request the documents:

#import json

url = 'https://www.ercot.com/mp/data-products/data-product-details?id=NP3-233-CD'
reqUrl, ddUrl = getReqUrl(url)
reqRes = requests.get(reqUrl[0]).text
rsJson = json.loads(reqRes)

for doc in rsJson['ListDocsByRptTypeRes']['DocumentList']:
  d = doc['Document'] 
  downloadLink = ddUrl+d['DocID']
  #print(f"{d['FriendlyName']} {d['PublishDate']} {downloadLink}")
  print(f"Download '{d['ConstructedName']}' at\n\t {downloadLink}")
print(len(rsJson['ListDocsByRptTypeRes']['DocumentList']))

The print results will look like