IndexError: list index out of range while scraping tables to excel issues

Question

I have a few issues I'm trying to resolve. First, I'm trying to figure out how I can get the "International Regular Season Stats - Per Game" and "International Regular Season Stats - Advanced Stats" tables of each player. Right now I'm trying to work with the international, but ultimately, I want both next to each other. However, the table number for each player could be different so Carlos Delfino is [18] but Fabricio Oberto has it as [11]. The error I'm getting because of this is '''IndexError: list index out of range``` Also, i'd like to put the player's name on each row for every season he's played. I also want to leave a blank row for a player without the table (Yao Ming).

import requests
from bs4 import BeautifulSoup
playernames=['Carlos Delfino', 'Fabricio Oberto', 'Yao Ming']

for name in playernames:
  fname=name.split(" ")[0]
  lname=name.split(" ")[1]
  url="https://basketball.realgm.com/search?q={}+{}".format(fname,lname)
  response = requests.get(url)

  soup = BeautifulSoup(response.content, 'html.parser')
  table = (soup.find_all('table', attrs={'class': 'tablesaw', 'data-tablesaw-mode-exclude': 'columntoggle'})[18]).find_next('tbody')
  print(table)  

  columns = ['Season', 'Team', 'League', 'GP', 'GS', 'TS%', 'eFG%', 'ORB%', 'DRB%', 'TRB%', 'AST%', 'TOV%', 'STL%', 'BLK%', 'USG%', 'Total S%', 'PPR', 'PPS', 'ORtg', 'DRtg', 'PER']
  df = pd.DataFrame(columns=columns)

  trs = table.find_all('tr')
  for tr in trs:
    tds = tr.find_all('td')
    row = [td.text.replace('\n', '') for td in tds]
    df = df.append(pd.Series(row, index=columns), ignore_index=True)

df.to_csv('international players.csv', index=False) ```

Answer 1

You can find the tags that have that specific header text (which is the <h2> tag), then use .findNext() to grab the table that follows. I would also consider using pandas' .read_html() , as it uses beautifulsoup under the hood to parse <table> tags and does the hardwork of iterating through the <th> , <tr> , and < td> tags. Once you have those tables, merge them on the common columns and append them to a final dataframe.

To handle guys like Yao Ming who doesn't have international tables, you can use the try/except . So essentially it'll try to grab the international table, if it's not there, it'll make a dataframe with just Yao's name and append that, leaving the row empty:

import requests
from bs4 import BeautifulSoup
import pandas as pd


playernames=['Carlos Delfino', 'Fabricio Oberto', 'Yao Ming']

result = pd.DataFrame()
for name in playernames:

    fname=name.split(" ")[0]
    lname=name.split(" ")[1]
    url="https://basketball.realgm.com/search?q={}+{}".format(fname,lname)
    response = requests.get(url)

    soup = BeautifulSoup(response.content, 'html.parser')


    try:
        table1 = soup.find('h2',text='International Regular Season Stats - Per Game').findNext('table')
        table2 = soup.find('h2',text='International Regular Season Stats - Advanced Stats').findNext('table')

        df1 = pd.read_html(str(table1))[0]
        df2 = pd.read_html(str(table2))[0]

        commonCols = list(set(df1.columns) & set(df2.columns))
        df = df1.merge(df2, how='left', on=commonCols)
        df['Player'] = name

    except:
        print ('No international table for %s.' %name)
        df = pd.DataFrame([name], columns=['Player'])

    result = result.append(df, sort=False).reset_index(drop=True)

cols = list(result.columns)
cols = [cols[-1]] + cols[:-1]
result = result[cols]
result.to_csv('international players.csv', index=False)

Output:

print (result.to_string())
       Season                               Team        League    GP    GS   MIN   FGM    FGA    FG%   3PM   3PA    3P%   FTM   FTA    FT%   OFF   DEF   TRB   AST   STL   BLK    PF   TOV    PTS    TS%   eFG%   ORB%   DRB%   TRB%   AST%   TOV%  STL%  BLK%   USG%  Total S %    PPR   PPS   ORtg   DRtg    PER           Player
0     2002-03         Fortituto Kontatto Bologna    Euroleague  16.0  15.0  31.8  4.25  10.81  0.393  1.44  4.94  0.291  2.06  3.06  0.673  1.12  5.94  7.06  1.69  1.69  0.31  2.94  1.38  12.00  0.493  0.460   4.52  23.74  14.15  10.12  10.16  2.92  1.01  20.30     135.77  -0.77  1.11  105.0  102.2  15.38   Carlos Delfino
1   2003-04 *                          All Teams   All Leagues  62.0  48.0  28.7  4.06   9.74  0.417  1.27  3.89  0.328  2.21  3.02  0.733  1.31  4.26  5.56  1.73  1.90  0.13  2.10  1.77  11.61  0.525  0.483   5.96  19.29  12.65  10.33  13.81  3.62  0.50  21.08     147.76  -2.15  1.19  107.5  109.2  16.08   Carlos Delfino
2   2003-04 *         Fortituto Kontatto Bologna        Lega A  41.0  32.0  27.9  3.90   9.32  0.419  1.27  3.93  0.323  2.15  3.02  0.710  1.07  4.27  5.34  1.46  2.02  0.12  1.76  1.80  11.22  0.527  0.487   5.09  19.65  12.48   8.89  14.49  3.91  0.49  20.90     145.15  -3.01  1.20  105.7  106.0  15.15   Carlos Delfino
3   2003-04 *         Fortituto Kontatto Bologna    Euroleague  21.0  16.0  30.4  4.38  10.57  0.414  1.29  3.81  0.338  2.33  3.00  0.778  1.76  4.24  6.00  2.24  1.67  0.14  2.76  1.71  12.38  0.521  0.475   7.49  18.64  12.97  13.03  12.60  3.09  0.51  21.43     152.97  -0.71  1.17  110.7  115.3  16.58   Carlos Delfino
4     2008-09                             Khimki       Eurocup  10.0   7.0  27.6  4.50  10.40  0.433  2.00  4.80  0.417  2.00  2.70  0.741  0.70  2.90  3.60  2.60  1.50  0.10  1.80  1.50  13.00  0.561  0.529   3.08  15.14   8.60  18.80  11.46  3.10  0.44  22.74     159.01   0.86  1.25  118.0   98.6  19.48   Carlos Delfino
5     2016-17                       Boca Juniors        Liga A  13.0   5.0  23.7  3.69   9.46  0.390  1.00  4.31  0.232  2.77  3.62  0.766  0.77  3.15  3.92  2.23  1.15  0.15  1.54  2.08  11.15  0.505  0.443   4.03  15.45   9.93  14.53  15.82  2.55  0.60  25.67     138.83  -2.44  1.18   98.5  110.6  14.82   Carlos Delfino
6   2017-18 *                          All Teams   All Leagues   6.0   0.0  10.3  0.50   2.67  0.188  0.50  2.00  0.250  0.33  0.33  1.000  0.33  1.67  2.00  0.33  0.17  0.00  1.00  1.00   1.83  0.326  0.281   3.91  20.06  11.88   5.15  26.22  0.88  0.00  17.81     143.75  -7.51  0.69   59.2  111.7  -0.76   Carlos Delfino
7   2017-18 *  KIROLBET Baskonia Vitoria-Gasteiz           ACB   4.0   0.0  12.9  0.75   3.75  0.200  0.75  2.75  0.273  0.50  0.50  1.000  0.50  2.00  2.50  0.50  0.25  0.00  1.50  1.25   2.75  0.346  0.300   4.40  18.93  11.40   5.99  23.95  1.01  0.00  18.52     147.27  -6.73  0.73   65.1  113.0   0.21   Carlos Delfino
8   2017-18 *  KIROLBET Baskonia Vitoria-Gasteiz    Euroleague   2.0   0.0   5.0  0.00   0.50  0.000  0.00  0.50  0.000  0.00  0.00  0.000  0.00  1.00  1.00  0.00  0.00  0.00  0.00  0.50   0.00  0.000  0.000   0.00  25.40  13.22   0.00  50.00  0.00  0.00  10.64       0.00 -10.21  0.00    0.0  109.3  -6.67   Carlos Delfino
9   2018-19 *                          All Teams   All Leagues  23.0   7.0  18.8  2.48   5.57  0.445  1.35  3.39  0.397  1.65  2.04  0.809  0.43  2.57  3.00  1.30  0.87  0.13  1.48  1.09   7.96  0.615  0.566   2.86  15.85   9.56  11.83  14.39  2.50  0.73  19.37     165.13  -1.17  1.43  113.5  109.6  15.37   Carlos Delfino
10  2018-19 *                        Fiat Torino        Lega A  10.0   4.0  19.3  2.20   5.40  0.407  1.50  3.70  0.405  1.80  2.40  0.750  0.20  3.00  3.20  1.20  0.70  0.10  1.40  1.10   7.70  0.596  0.546   1.31  17.44   9.86  10.54  14.56  1.95  0.57  18.80     156.28  -1.57  1.43  110.2  110.6  13.67   Carlos Delfino
11  2018-19 *         Fortituto Kontatto Bologna  Legadue Gold   7.0   0.0  17.1  3.14   6.14  0.512  1.43  3.14  0.455  1.43  1.71  0.833  0.43  2.29  2.71  1.86  1.29  0.14  1.43  1.14   9.14  0.663  0.628   2.97  15.82   9.39  18.09  14.21  3.97  0.81  22.38     179.95   0.54  1.49  120.1  100.5  22.76   Carlos Delfino
12  2018-19 *                        Fiat Torino       Eurocup   6.0   3.0  19.7  2.17   5.17  0.419  1.00  3.17  0.316  1.67  1.83  0.909  0.83  2.17  3.00  0.83  0.67  0.17  1.67  1.00   7.00  0.586  0.516   5.24  13.10   9.25   7.47  14.34  1.87  0.91  17.22     164.42  -2.22  1.35  110.4  118.1  12.78   Carlos Delfino
13    2000-01  KIROLBET Baskonia Vitoria-Gasteiz    Euroleague  22.0  18.0  28.0  4.77   7.27  0.656  0.00  0.00  0.000  1.36  3.00  0.455  3.09  4.27  7.36  1.09  2.00  0.59  3.36  1.27  10.91  0.635  0.656  14.56  21.15  17.77   7.41  12.90  4.05  2.36  17.27     111.08  -2.02  1.50  125.2   96.9  22.89  Fabricio Oberto
14    2001-02  KIROLBET Baskonia Vitoria-Gasteiz    Euroleague  11.0  10.0  25.2  3.91   8.00  0.489  0.00  0.09  0.000  1.18  2.64  0.448  2.55  3.91  6.45  1.55  1.09  0.73  4.18  2.18   9.00  0.491  0.489  13.88  20.81  17.39  10.43  19.24  2.32  2.86  20.96      93.69  -4.50  1.12   97.5  101.9  13.34  Fabricio Oberto
15    2002-03                    Valencia Basket       Eurocup  18.0  16.0  26.4  5.67   8.83  0.642  0.00  0.00  0.000  1.44  2.67  0.542  2.06  2.72  4.78  1.72  1.00  1.33  3.17  2.06  12.78  0.638  0.642  12.18  14.05  13.18  12.17  17.04  2.16  5.47  22.37     118.32  -3.57  1.45  119.5  105.4  20.72  Fabricio Oberto
16  2003-04 *                          All Teams   All Leagues  57.0  49.0  26.5  5.39   8.86  0.608  0.00  0.02  0.000  1.32  3.05  0.431  1.47  4.07  5.54  1.91  0.86  0.70  3.39  1.88  12.09  0.592  0.608   7.50  21.43  14.34  13.44  15.54  1.82  2.82  21.88     103.90  -2.31  1.36  112.0  107.6  17.36  Fabricio Oberto
17  2003-04 *                    Valencia Basket           ACB  39.0  33.0  26.2  5.15   8.64  0.596  0.00  0.03  0.000  1.21  2.74  0.439  1.36  4.15  5.51  1.69  0.90  0.74  3.38  1.82  11.51  0.585  0.596   6.88  21.76  14.20  11.90  15.60  1.91  2.98  21.34     103.57  -2.66  1.33  110.0  106.0  17.34  Fabricio Oberto
18  2003-04 *                    Valencia Basket    Euroleague  18.0  16.0  27.3  5.89   9.33  0.631  0.00  0.00  0.000  1.56  3.72  0.418  1.72  3.89  5.61  2.39  0.78  0.61  3.39  2.00  13.33  0.608  0.631   8.84  20.67  14.65  16.73  15.42  1.62  2.47  22.97     104.89  -1.48  1.43  115.9  110.9  19.80  Fabricio Oberto
19  2004-05 *                          All Teams   All Leagues  46.0  42.0  28.3  6.11   9.61  0.636  0.00  0.02  0.000  1.37  3.30  0.414  2.20  4.98  7.17  2.04  1.07  0.54  3.52  2.11  13.59  0.614  0.636  11.46  22.36  17.32  13.85  16.01  2.12  1.93  23.05     105.02  -2.69  1.41  115.9  108.3  20.20  Fabricio Oberto
20  2004-05 *                    Valencia Basket           ACB  30.0  26.0  28.5  6.37   9.73  0.654  0.00  0.00  0.000  1.43  3.17  0.453  2.30  4.97  7.27  2.13  1.27  0.50  3.90  2.03  14.17  0.637  0.654  11.91  22.40  17.51  13.87  15.45  2.49  1.70  22.77     110.67  -2.14  1.46  120.6  112.1  21.81  Fabricio Oberto
21  2004-05 *                    Valencia Basket       Eurocup  16.0  16.0  27.9  5.62   9.38  0.600  0.00  0.06  0.000  1.25  3.56  0.351  2.00  5.00  7.00  1.88  0.69  0.62  2.81  2.25  12.50  0.571  0.600  10.60  22.29  16.95  13.80  17.06  1.40  2.41  23.58      95.09  -3.64  1.33  107.0  101.1  18.29  Fabricio Oberto
22    2012-13        Asociacion Deportiva Atenas        Liga A  13.0  11.0  23.5  4.00   7.31  0.547  0.00  0.00  0.000  1.23  2.31  0.533  1.00  4.08  5.08  2.54  0.31  0.15  3.31  1.15   9.23  0.555  0.547   5.05  22.20  13.30  23.04  12.18  0.73  0.69  19.56     108.07   2.32  1.26  112.4  104.5  15.97  Fabricio Oberto
23        NaN                                NaN           NaN   NaN   NaN   NaN   NaN    NaN    NaN   NaN   NaN    NaN   NaN   NaN    NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN    NaN    NaN    NaN    NaN    NaN    NaN    NaN    NaN   NaN   NaN    NaN        NaN    NaN   NaN    NaN    NaN    NaN         Yao Ming