I keep getting a list index out of range error when trying to run this code, the code parses a sites tables by going through its pages and inputs the data into an excel sheet.
The error is being given on revenue = cols[0].string:
from urllib.request import urlopen
from bs4 import BeautifulSoup
from openpyxl import Workbook
from openpyxl.cell import get_column_letter
import datetime
now = datetime.datetime.now()
wb = Workbook()
dest_filename = r'iOS Top Grossing Data.xlsx'
ws = wb.active
ws = wb.create_sheet()
ws.title = now.strftime("%m-%d-%y")
sh = wb.get_sheet_by_name('Sheet')
wb.remove_sheet(sh)
ws['A1'] = "REVENUE"
ws.column_dimensions['A'].width = 11
ws.cell('A1').style.alignment.horizontal = 'center'
ws.cell('A1').style.font.bold = True
ws['B1'] = "FREE"
ws.column_dimensions['B'].width = 7
ws.cell('B1').style.alignment.horizontal = 'center'
ws.cell('B1').style.font.bold = True
ws['C1'] = "PAID"
ws.column_dimensions['C'].width = 7
ws.cell('C1').style.alignment.horizontal = 'center'
ws.cell('C1').style.font.bold = True
ws['D1'] = "GAME"
ws.column_dimensions['D'].width = 27
ws.cell('D1').style.alignment.horizontal = 'center'
ws.cell('D1').style.font.bold = True
ws['E1'] = "PRICE"
ws.column_dimensions['E'].width = 7
ws.cell('E1').style.alignment.horizontal = 'center'
ws.cell('E1').style.font.bold = True
ws['F1'] = "REVENUE"
ws.column_dimensions['F'].width = 11
ws.cell('F1').style.alignment.horizontal = 'center'
ws.cell('F1').style.font.bold = True
ws['G1'] = "ARPU INDEX"
ws.column_dimensions['G'].width = 15
ws.cell('G1').style.alignment.horizontal = 'center'
ws.cell('G1').style.font.bold = True
ws['H1'] = "DAILY NEW USERS"
ws.column_dimensions['H'].width = 17
ws.cell('H1').style.alignment.horizontal = 'center'
ws.cell('H1').style.font.bold = True
ws['I1'] = "DAILY ACTIVE USERS"
ws.column_dimensions['I'].width = 19
ws.cell('I1').style.alignment.horizontal = 'center'
ws.cell('I1').style.font.bold = True
ws['J1'] = "ARPU"
ws.column_dimensions['J'].width = 7
ws.cell('J1').style.alignment.horizontal = 'center'
ws.cell('J1').style.font.bold = True
ws['K1'] = "RANK CHANGE"
ws.column_dimensions['K'].width = 14
ws.cell('K1').style.alignment.horizontal = 'center'
ws.cell('K1').style.font.bold = True
page = 0
while page < 6:
page += 1
url = "http://thinkgaming.com/app-sales-data/?page=" + str(page)
html = str(urlopen(url).read())
soup = BeautifulSoup(html)
table = soup.find("table")
counter = 0
while counter < 51:
rows = table.findAll('tr')[counter]
cols = rows.findAll('td')
revenue = cols[0].string
revenue = revenue.replace('\\n', '')
revenue = revenue.replace(' ', '')
free = cols[1].string
free = free.replace('\\n', '')
free = free.replace(' ', '')
paid = cols[2].string
paid = paid.replace('\\n', '')
paid = paid.replace(' ', '')
game = cols[3].string
price = cols[4].string
price = price.replace('\\n', '')
price = price.replace(' ', '')
revenue2 = cols[5].string
revenue2 = revenue2.replace('\\n', '')
revenue2 = revenue2.replace(' ', '')
dailynewusers = cols[6].string
dailynewusers = dailynewusers.replace('\\n', '')
dailynewusers = dailynewusers.replace(' ', '')
cell_location = counter
cell_location += 1
ws['A'+str(cell_location)] = revenue
counter += 1
wb.save(filename = dest_filename)
Here's the traceback:
Traceback (most recent call last):
File "C:\Users\shiver_admin\Desktop\script.py", line 89, in <module> revenue = cols[0].string IndexError: list index out of range
Same as the comments, you are not getting any <td>
tags simply because they don't exist, especially for index [0]
. The first <tr>
tag in that table is this:
If you notice, it has headers inside. Basically, you should start your counter
at 1 instead of 0.
Another way to be sure that you're getting the correct rows is checking if they have classes. If you noticed, the proper <tr>
rows have classes inside them ( odd
and even
). You can employ something like table.find_all("tr", class_=True)
to get them.
Sample code (note: coded in Python 2.7, but easy enough to modify to suit Python 3.x):
import requests as rq
from bs4 import BeautifulSoup as bsoup
url = "http://thinkgaming.com/app-sales-data/?page=1"
r = rq.get(url)
soup = bsoup(r.content)
table = soup.find("table", class_="table")
rows = table.find_all("tr", class_=True)
cols = [td.get_text().strip().encode("utf-8") for td in rows[0].find_all("td")]
print cols
Result:
['1', '10', '-', 'Clash of Clans', 'Free', 'n/a', '44,259']
[Finished in 2.8s]
Let us know if this helps.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.