[英]Scraping data from tables using python 2.7
I have tried this two different ways and cannot get either to work. 我尝试了两种不同的方法,但都无法正常工作。 I am trying to scrape the stats on this webpage: http://www.cbssports.com/nfl/gametracker/boxscore/NFL_20161120_JAC@DET/ within "TEAM STATS". 我正在尝试在此页面上收集统计信息: http : //www.cbssports.com/nfl/gametracker/boxscore/NFL_20161120_JAC@DET/在“团队统计”中。 I need the numbers after a specified stat category ex: "NET YARDS RUSHING". 我需要指定统计信息类别后的数字,例如:“ NET YARDS RUSHING”。 Below is what I have tried without any success. 以下是我尝试没有成功的尝试。
First way: 第一种方式:
import pickle
import math
import os
import urllib2
from lxml import etree
from bs4 import BeautifulSoup
from urllib import urlopen
from openpyxl import load_workbook
from openpyxl import Workbook
from openpyxl.styles import Color, PatternFill, Font, Border
from openpyxl.styles import colors
from openpyxl.cell import Cell
Last Two Game info Home [H] or Away [A]
favLastGM = 'H' #Higher week number 2
favLastGM2 = 'A' #Lower week number 1
#Game Info (Favorite) Last Game Played - CBS Sports (Change Every Week)
favPrevGMInfoUrl = 'http://www.cbssports.com/nfl/gametracker/boxscore/NFL_20161120_JAC@DET/'
response8 = urllib2.urlopen(favPrevGMInfoUrl)
htmlparser8 = etree.HTMLParser()
tree8 = etree.parse(response8,htmlparser8)
#FAVORITE
if favLastGM == 'A': #This Gives Opposite of Away Team Net Rushing Yards - SO HOME Net Rushing Yards
text = tree8.xpath('//td[contains(text(),"Net Yards Rushing")]/parent::td/following-sibling::td[1]/text()')
if text:
favDef_rushYards_L2_1 = int(text[0].strip())
print("test"),
print favDef_rushYards_L2_1
print ("Enter: Total Rushing Yards Allowed from Favored Team Defense for last game played: "),
print favDef_rushYards_L2_1
elif favLastGM == 'H': #This Gives Opposite of Home Team Net Rushing Yards - SO AWAY Net Rushing Yards
text = tree8.xpath('//td[contains(text(),"Net Yards Rushing")]/parent::td/following-sibling::td[0]/text()')
if text:
favDef_rushYards_L2_1 = int(text[0].strip())
print("test"),
print favDef_rushYards_L2_1
print ("Enter: Total Rushing Yards Allowed from Favored Team Defense for last game played: "),
print favDef_rushYards_L2_1
else:
print("***************************************************")
print("NOT A VALID ENTRY - favLastGM !")
print("***************************************************")
Second way: 第二种方式:
import pickle
import math
import os
import urllib2
from lxml import etree
from bs4 import BeautifulSoup
from urllib import urlopen
from openpyxl import load_workbook
from openpyxl import Workbook
from openpyxl.styles import Color, PatternFill, Font, Border
from openpyxl.styles import colors
from openpyxl.cell import Cell
#Last Two Game info Home [H] or Away [A]
favLastGM = 'H' #Higher week number 2
favLastGM2 = 'A' #Lower week number 1
#Game Info (Favorite) Last Game Played - CBS Sports (Change Every Week)
favPrevGMInfoUrl = 'http://www.cbssports.com/nfl/gametracker/boxscore/NFL_20161120_JAC@DET/'
favPrevGMhtml2 = urlopen(favPrevGMInfoUrl).read()
favPrevGMsoup2 = BeautifulSoup(favPrevGMhtml2)
favPrevGM2Reg = favPrevGMsoup2.find("table", { "class" : "team-stats" })
favPrevGM2Reg2 = []
if favLastGM == 'A': #This Gives Opposite of Away Team Net Rushing Yards - SO HOME Net Rushing Yards
rush = 'Net Yards Rushing'
for row in favPrevGM2Reg.findAll("tr"):
if rush in row.findNext('td'): #Change Year for every new season
for item in row.findAll("td"):
favPrevGM2Reg.append(item.text)
favDef_rushYards_L2_1 = float(favPrevGM2Reg[1])
print ("Enter: Total Rushing Yards Allowed from Favored Team Defense for last game played: "),
print favDef_rushYards_L2_1
elif favLastGM == 'H': #This Gives Opposite of Home Team Net Rushing Yards - SO AWAY Net Rushing Yards
rush = 'Net Yards Rushing'
for row in favPrevGM2Reg.findAll("tr"):
if rush in row.findNext('td'): #Change Year for every new season
for item in row.findAll("td"):
favPrevGM2Reg.append(item.text)
favDef_rushYards_L2_1 = float(favPrevGM2Reg[0])
print ("Enter: Total Rushing Yards Allowed from Favored Team Defense for last game played: "),
print favDef_rushYards_L2_1
else:
print("***************************************************")
print("NOT A VALID ENTRY - favLastGM !")
print("***************************************************")
You are looking for the xpath: 您正在寻找xpath:
//td[contains(text(),"Net Yards Rushing")]/following-sibling::td
What this does is select the beginning td, which you did right, but you want its siblings, not its parents' siblings, so you need to add following-sibling::td directly after the td. 这样做是选择正确的开始td,但是您希望它的同级而不是其父母的同级,因此您需要在td之后直接添加following-sibling :: td。 This will give you 2 results in order of appearance in the table 这将按照表格中的出现顺序为您提供2个结果
I didn't write out the whole code, but these two lines will give you the home and away rushing yards. 我没有写出完整的代码,但是这两行代码将为您提供家常便饭的地方。
import urllib2
from lxml import etree
favPrevGMInfoUrl = 'http://www.cbssports.com/nfl/gametracker/boxscore/NFL_20161120_JAC@DET/'
response8 = urllib2.urlopen(favPrevGMInfoUrl)
htmlparser8 = etree.HTMLParser()
tree8 = etree.parse(response8,htmlparser8)
away = tree8.xpath('//tr[@data-category="rushing_yards"]//td[@class="stat-value away"]/text()')
home = tree8.xpath('//tr[@data-category="rushing_yards"]//td[@class="stat-value home"]/text()')
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.