如何使用pd.read_html从数据框中剥离列并将输出作为列表返回

Question

我正在尝试使用Pandas read_html函数获取股票代码列表（而不是使用Beautiful Soup来抓取网页）。

我引用的网站是：

https://en.wikipedia.org/wiki/List_of_S%26P_500_companies

所需的输出是：

['MMM', 'ABT', 'ABBV', 'ACN', 'ATVI' ... ] 

我的代码是：

import pandas as pd

url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'
df = pd.read_html(url)[0]
#df.columns = df.iloc[0]
df.drop(df.index[0], inplace=True)
tickers = df['Symbol'].tolist()

此代码的输出是一个数据框，如下所示：

df.head()

    Symbol  Security    SEC filings GICS Sector GICS Sub Industry   Headquarters Location   Date first added    CIK Founded
1   ABT Abbott Laboratories reports Health Care Health Care Equipment   North Chicago, Illinois 1964-03-31  1800    1888
2   ABBV    AbbVie Inc. reports Health Care Pharmaceuticals North Chicago, Illinois 2012-12-31  1551152 2013 (1888)
3   ABMD    ABIOMED Inc reports Health Care Health Care Equipment   Danvers, Massachusetts  2018-05-31  815094  1981
4   ACN Accenture plc   reports Information Technology  IT Consulting & Other Services  Dublin, Ireland 2011-07-06  1467373 1989
5   ATVI    Activision Blizzard reports Communication Services  Interactive Home Entertainment  Santa Monica, California    2015-08-31  718877  2008

如果我取消注释df.columns = df.iloc[0] ，那么Pandas会抛出以下错误消息

KeyError: 'Symbol'

行df.iloc[0]返回：

Symbol                                       ABT
Security                     Abbott Laboratories
SEC filings                              reports
GICS Sector                          Health Care
GICS Sub Industry          Health Care Equipment
Headquarters Location    North Chicago, Illinois
Date first added                      1964-03-31
CIK                                         1800
Founded                                     1888

这不是我正在寻找的（而是在包含'Symbol'列的那个之前的标题行）。

有谁看到我在这里做错了什么？ 谢谢！

Answer 1

使用pandas库来读取html表数据。 tolist（）用于将系列转换为列表。

import pandas as pd

url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'
df = pd.read_html(url)[0]
# filter table data by selected columns
df = df[['Symbol']]
tickers = df['Symbol'].tolist()

print(tickers)

O / P：

['MMM', 'ABT', 'ABBV', 'ABMD', 'ACN', 'ATVI', 'ADBE', 'AMD', 'AAP', 'AES', 'AMG', 'AFL', 'A', 'APD', 'AKAM', 'ALK', 'ALB', 'ARE', 'ALXN', 'ALGN', 'ALLE', 'AGN', 'ADS', 'LNT', 'ALL', 'GOOGL', 'GOOG', 'MO', 'AMZN', 'AMCR', 'AEE', 'AAL', 'AEP', 'AXP', 'AIG', 'AMT', 'AWK', 'AMP', 'ABC', 'AME', 'AMGN', 'APH', 'APC', 'ADI', 'ANSS', 'ANTM', 'AON', 'AOS', 'APA', 'AIV', 'AAPL', 'AMAT', 'APTV', 'ADM', 'ARNC', 'ANET', 'AJG', 'AIZ', 'ATO', 'T', 'ADSK', 'ADP', 'AZO', 'AVB', 'AVY', 'BHGE', 'BLL', 'BAC', 'BK', 'BAX', 'BBT', 'BDX', 'BRK.B', 'BBY', 'BIIB', 'BLK', 'HRB', 'BA', 'BKNG', 'BWA', 'BXP', 'BSX', 'BMY', 'AVGO', 'BR', 'BF.B', 'CHRW', 'COG', 'CDNS', 'CPB', 'COF', 'CPRI', 'CAH', 'KMX', 'CCL', 'CAT', 'CBOE', 'CBRE', 'CBS', 'CE', 'CELG', 'CNC', 'CNP', 'CTL', 'CERN', 'CF', 'SCHW', 'CHTR', 'CVX', 'CMG', 'CB', 'CHD', 'CI', 'XEC', 'CINF', 'CTAS', 'CSCO', 'C', 'CFG', 'CTXS', 'CLX', 'CME', 'CMS', 'KO', 'CTSH', 'CL', 'CMCSA', 'CMA', 'CAG', 'CXO', 'COP', 'ED', 'STZ', 'COO', 'CPRT', 'GLW', 'CTVA', 'COST', 'COTY', 'CCI', 'CSX', 'CMI', 'CVS', 'DHI', 'DHR', 'DRI', 'DVA', 'DE', 'DAL', 'XRAY', 'DVN', 'FANG', 'DLR', 'DFS', 'DISCA', 'DISCK', 'DISH', 'DG', 'DLTR', 'D', 'DOV', 'DOW', 'DTE', 'DUK', 'DRE', 'DD', 'DXC', 'ETFC', 'EMN', 'ETN', 'EBAY', 'ECL', 'EIX', 'EW', 'EA', 'EMR', 'ETR', 'EOG', 'EFX', 'EQIX', 'EQR', 'ESS', 'EL', 'EVRG', 'ES', 'RE', 'EXC', 'EXPE', 'EXPD', 'EXR', 'XOM', 'FFIV', 'FB', 'FAST', 'FRT', 'FDX', 'FIS', 'FITB', 'FE', 'FRC', 'FISV', 'FLT', 'FLIR', 'FLS', 'FMC', 'FL', 'F', 'FTNT', 'FTV', 'FBHS', 'FOXA', 'FOX', 'BEN', 'FCX', 'GPS', 'GRMN', 'IT', 'GD', 'GE', 'GIS', 'GM', 'GPC', 'GILD', 'GPN', 'GS', 'GWW', 'HAL', 'HBI', 'HOG', 'HIG', 'HAS', 'HCA', 'HCP', 'HP', 'HSIC', 'HSY', 'HES', 'HPE', 'HLT', 'HFC', 'HOLX', 'HD', 'HON', 'HRL', 'HST', 'HPQ', 'HUM', 'HBAN', 'HII', 'IDXX', 'INFO', 'ITW', 'ILMN', 'IR', 'INTC', 'ICE', 'IBM', 'INCY', 'IP', 'IPG', 'IFF', 'INTU', 'ISRG', 'IVZ', 'IPGP', 'IQV', 'IRM', 'JKHY', 'JEC', 'JBHT', 'JEF', 'SJM', 'JNJ', 'JCI', 'JPM', 'JNPR', 'KSU', 'K', 'KEY', 'KEYS', 'KMB', 'KIM', 'KMI', 'KLAC', 'KSS', 'KHC', 'KR', 'LB', 'LHX', 'LH', 'LRCX', 'LW', 'LEG', 'LEN', 'LLY', 'LNC', 'LIN', 'LKQ', 'LMT', 'L', 'LOW', 'LYB', 'MTB', 'MAC', 'M', 'MRO', 'MPC', 'MKTX', 'MAR', 'MMC', 'MLM', 'MAS', 'MA', 'MKC', 'MXIM', 'MCD', 'MCK', 'MDT', 'MRK', 'MET', 'MTD', 'MGM', 'MCHP', 'MU', 'MSFT', 'MAA', 'MHK', 'TAP', 'MDLZ', 'MNST', 'MCO', 'MS', 'MOS', 'MSI', 'MSCI', 'MYL', 'NDAQ', 'NOV', 'NKTR', 'NTAP', 'NFLX', 'NWL', 'NEM', 'NWSA', 'NWS', 'NEE', 'NLSN', 'NKE', 'NI', 'NBL', 'JWN', 'NSC', 'NTRS', 'NOC', 'NCLH', 'NRG', 'NUE', 'NVDA', 'ORLY', 'OXY', 'OMC', 'OKE', 'ORCL', 'PCAR', 'PKG', 'PH', 'PAYX', 'PYPL', 'PNR', 'PBCT', 'PEP', 'PKI', 'PRGO', 'PFE', 'PM', 'PSX', 'PNW', 'PXD', 'PNC', 'PPG', 'PPL', 'PFG', 'PG', 'PGR', 'PLD', 'PRU', 'PEG', 'PSA', 'PHM', 'PVH', 'QRVO', 'PWR', 'QCOM', 'DGX', 'RL', 'RJF', 'RTN', 'O', 'RHT', 'REG', 'REGN', 'RF', 'RSG', 'RMD', 'RHI', 'ROK', 'ROL', 'ROP', 'ROST', 'RCL', 'CRM', 'SBAC', 'SLB', 'STX', 'SEE', 'SRE', 'SHW', 'SPG', 'SWKS', 'SLG', 'SNA', 'SO', 'LUV', 'SPGI', 'SWK', 'SBUX', 'STT', 'SYK', 'STI', 'SIVB', 'SYMC', 'SYF', 'SNPS', 'SYY', 'TROW', 'TTWO', 'TPR', 'TGT', 'TEL', 'FTI', 'TFX', 'TXN', 'TXT', 'TMO', 'TIF', 'TWTR', 'TJX', 'TMK', 'TSS', 'TSCO', 'TDG', 'TRV', 'TRIP', 'TSN', 'UDR', 'ULTA', 'USB', 'UAA', 'UA', 'UNP', 'UAL', 'UNH', 'UPS', 'URI', 'UTX', 'UHS', 'UNM', 'VFC', 'VLO', 'VAR', 'VTR', 'VRSN', 'VRSK', 'VZ', 'VRTX', 'VIAB', 'V', 'VNO', 'VMC', 'WAB', 'WMT', 'WBA', 'DIS', 'WM', 'WAT', 'WEC', 'WCG', 'WFC', 'WELL', 'WDC', 'WU', 'WRK', 'WY', 'WHR', 'WMB', 'WLTW', 'WYNN', 'XEL', 'XRX', 'XLNX', 'XYL', 'YUM', 'ZBH', 'ZION', 'ZTS']

Answer 2

问题是如果使用：

df.columns = df.iloc[0]

...然后按第一个数据行重写DataFrame的列，因此不存在原始的Symbol列和错误引发：

url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'
df = pd.read_html(url)[0]
print (df.head(3))
  Symbol             Security SEC filings  GICS Sector  \
0    MMM           3M Company     reports  Industrials   
1    ABT  Abbott Laboratories     reports  Health Care   
2   ABBV          AbbVie Inc.     reports  Health Care   

          GICS Sub Industry    Headquarters Location Date first added  \
0  Industrial Conglomerates      St. Paul, Minnesota              NaN   
1     Health Care Equipment  North Chicago, Illinois       1964-03-31   
2           Pharmaceuticals  North Chicago, Illinois       2012-12-31   

       CIK      Founded  
0    66740         1902  
1     1800         1888  
2  1551152  2013 (1888) 

print (df.columns)
Index(['Symbol', 'Security', 'SEC filings', 'GICS Sector', 'GICS Sub Industry',
       'Headquarters Location', 'Date first added', 'CIK', 'Founded'],
      dtype='object')

---

df.columns = df.iloc[0]
print (df.head(3))
0   MMM           3M Company  reports  Industrials  Industrial Conglomerates  \
0   MMM           3M Company  reports  Industrials  Industrial Conglomerates   
1   ABT  Abbott Laboratories  reports  Health Care     Health Care Equipment   
2  ABBV          AbbVie Inc.  reports  Health Care           Pharmaceuticals   

0      St. Paul, Minnesota         NaN    66740         1902  
0      St. Paul, Minnesota         NaN    66740         1902  
1  North Chicago, Illinois  1964-03-31     1800         1888  
2  North Chicago, Illinois  2012-12-31  1551152  2013 (1888) 

print (df.columns)
Index([                     'MMM',               '3M Company',
                        'reports',              'Industrials',
       'Industrial Conglomerates',      'St. Paul, Minnesota',
                              nan,                      66740,
                           '1902'],
      dtype='object', name=0)