从表中抓取数据
[英]web scraping data from tables
问题描述
I want to scrape the annual income statement, balance sheet and cash flow of this page. 
我要抓取该页面的年度损益表，资产负债表和现金流量。 https://www.google.com/finance?q=NYSE%3AIBM&fstype=ii&ei and put it into a dataframe. https://www.google.com/finance?q=NYSE%3AIBM&fstype=ii&ei ，并将其放入数据框。 As you can see you can change the data by clicking on different parts of the page. 如您所见，您可以通过单击页面的不同部分来更改数据。 Can someone show me how to scrape the annual income statement? 有人可以告诉我如何刮除年度损益表吗？ This what I have so far. 这是我到目前为止所拥有的。 I can see the data in the soup, but I don't know how to get to it. 我可以看到汤中的数据，但我不知道该怎么做。
from bs4 import BeautifulSoup
import requests
import pandas as pd
df =pd.DataFrame()
url = 'https://www.google.com/finance?q=NYSE%3AIBM&fstype=ii&ei'
headers = {'User-Agent': 'Mozilla/5.0 (Windows; Windows NT 6.1) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5'}
r = requests.get(url,headers=headers)
soup = BeautifulSoup(r.text,'html.parser')
2 个解决方案
解决方案1
2 已采纳 2016-03-10 12:28:11
why don't you just use pandas's read_html() function? 您为什么不只使用熊猫的read_html()函数？ As a result you get a list of data frames ( df ), one for each table that can be displayed by clicking on the options (among them the annual income statement): 结果，您将获得一个数据帧列表（ df ），每个表都可以通过单击选项（在年度收入表中）进行显示：
import pandas as pd
df = pd.read_html("https://www.google.com/finance?q=NYSE%3AIBM&fstype=ii&ei")
解决方案2
2 2016-03-10 15:07:10
The pandas read_html should so what you want but since you asked about bs4if you were to create the tables yourself it can be done quite easily with bs4: 大熊猫read_html应该是您想要的，但是由于您询问过bs4，如果要自己创建表，则可以使用bs4轻松完成：
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://www.google.com/finance?q=NYSE%3AIBM&fstype=ii&ei'
def get_rows(table):
# pull the data from each row, ignoring rows with no text
for row in table.select("tr"):
row = [x.text.strip() for x in row.select("td")]
if row:
yield row
def get_tables():
r = requests.get(url)
soup = BeautifulSoup(r.content)
# get all the tabes using the id selector
for table in soup.select("#fs-table"):
# create columns from th tags in the thead and get rows from helper
yield pd.DataFrame(list(get_rows(table)), columns=[x.text.strip() for x in table.find("thead").find_all("th")])
Which will give you six table, you can see the first if we call next(get_tables()) : 这将给您六个表，如果我们调用next(get_tables()) ，则可以看到第一个表：
In [4]: next(get_tables())
Out[4]:
In Millions of USD (except for per share items) \
0 Revenue
1 Other Revenue, Total
2 Total Revenue
3 Cost of Revenue, Total
4 Gross Profit
5 Selling/General/Admin. Expenses, Total
6 Research & Development
7 Depreciation/Amortization
8 Interest Expense(Income) - Net Operating
9 Unusual Expense (Income)
10 Other Operating Expenses, Total
11 Total Operating Expense
12 Operating Income
13 Interest Income(Expense), Net Non-Operating
14 Gain (Loss) on Sale of Assets
15 Other, Net
16 Income Before Tax
17 Income After Tax
18 Minority Interest
19 Equity In Affiliates
20 Net Income Before Extra. Items
21 Accounting Change
22 Discontinued Operations
23 Extraordinary Item
24 Net Income
25 Preferred Dividends
26 Income Available to Common Excl. Extra Items
27 Income Available to Common Incl. Extra Items
28 Basic Weighted Average Shares
29 Basic EPS Excluding Extraordinary Items
30 Basic EPS Including Extraordinary Items
31 Dilution Adjustment
32 Diluted Weighted Average Shares
33 Diluted EPS Excluding Extraordinary Items
34 Diluted EPS Including Extraordinary Items
35 Dividends per Share - Common Stock Primary Issue
36 Gross Dividends - Common Stock
37 Net Income after Stock Based Comp. Expense
38 Basic EPS after Stock Based Comp. Expense
39 Diluted EPS after Stock Based Comp. Expense
40 Depreciation, Supplemental
41 Total Special Items
42 Normalized Income Before Taxes
43 Effect of Special Items on Income Taxes
44 Income Taxes Ex. Impact of Special Items
45 Normalized Income After Taxes
46 Normalized Income Avail to Common
47 Basic Normalized EPS
48 Diluted Normalized EPS
3 months ending 2015-12-31 3 months ending 2015-09-30 \
0 22,059.00 19,280.00
1 - -
2 22,059.00 19,280.00
3 10,652.00 9,844.00
4 11,407.00 9,436.00
5 5,101.00 4,465.00
6 1,362.00 1,287.00
7 80.00 73.00
8 - -
9 12.00 112.00
10 -519.00 -32.00
11 16,962.00 15,659.00
12 5,097.00 3,621.00
13 - -
14 - -
15 2.00 -
16 5,099.00 3,621.00
17 4,461.00 2,962.00
18 - -
19 - -
20 4,461.00 2,962.00
21 - -
22 - -
23 - -
24 4,464.00 2,950.00
25 - -
26 4,460.00 2,962.00
27 4,463.00 2,950.00
28 - -
29 - -
30 - -
31 - 0.00
32 972.84 978.96
33 4.58 3.03
34 - -
35 1.30 1.30
36 - -
37 - -
38 - -
39 - -
40 - -
41 - -
42 - -
43 - -
44 - -
45 - -
46 - -
47 - -
48 4.87 2.98
3 months ending 2015-06-30 3 months ending 2015-03-31 \
0 20,813.00 19,590.00
1 - -
2 20,813.00 19,590.00
3 10,423.00 10,138.00
4 10,390.00 9,452.00
5 4,923.00 5,022.00
6 1,300.00 1,298.00
7 72.00 79.00
8 - -
9 178.00 285.00
10 -190.00 -20.00
11 16,716.00 16,762.00
12 4,097.00 2,828.00
13 - -
14 - -
15 127.00 173.00
16 4,224.00 3,001.00
17 3,526.00 2,416.00
18 - -
19 - -
20 3,526.00 2,416.00
21 - -
22 - -
23 - -
24 3,449.00 2,328.00
25 - -
26 3,526.00 2,416.00
27 3,449.00 2,328.00
28 - -
29 - -
30 - -
31 - -
32 986.70 992.30
33 3.57 2.43
34 - -
35 1.30 1.10
36 - -
37 - -
38 - -
39 - -
40 - -
41 - -
42 - -
43 - -
44 - -
45 - -
46 - -
47 - -
48 3.56 2.52
3 months ending 2014-12-31
0 24,113.00
1 -
2 24,113.00
3 11,251.00
4 12,862.00
5 5,375.00
6 1,320.00
7 93.00
8 -
9 578.00
10 -317.00
11 17,018.00
12 7,095.00
13 -
14 -
15 -
16 7,095.00
17 5,516.00
18 -
19 -
20 5,516.00
21 -
22 -
23 -
24 5,485.00
25 -
26 5,514.00
27 5,483.00
28 -
29 -
30 -
31 -
32 995.30
33 5.54
34 -
35 1.10
36 -
37 -
38 -
39 -
40 -
41 -
42 -
43 -
44 -
45 -
46 -
47 -
48 6.65
Or using lxml with xpaths: 或在xpaths中使用lxml ：
url = 'https://www.google.com/finance?q=NYSE%3AIBM&fstype=ii&ei'
from lxml.etree import fromstring, HTMLParser
import pandas as pd
def get_rows(table):
for row in table.xpath(".//tr"):
row = row.xpath("./td/text()")
if row:
yield row
def get_tables():
r = requests.get(url)
xml = fromstring(r.content, HTMLParser())
for table in xml.xpath("//table[@id='fs-table']"):
yield pd.DataFrame(list(get_rows(table)), columns=[x.strip() for x in table.xpath(".//th/text()")])
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.