BeautifulSoup Only Retrieves 1st element

Question

I'm currently trying to webscrap some website.

Here is part of my code:

import pandas as pd
from bs4 import BeautifulSoup
import requests


#Get the source code
r = requests.get('https://fbref.com/fr/comps/13/stats/Statistiques-Ligue-1#all_stats_standard')
c = r.content
soup = BeautifulSoup(c, "html.parser")
print(soup.prettify())

for a in soup:
    print (soup.find("td", {"data-stat" : "avg_age"}).text)

Basically, I have the whole source code inside "soup". However, when I call elements such as "td", {"data-stat": "avg_age"} I only get repeated result of the first row {"data-row":"0"} as an output:

29.1
29.1
29.1
29.1
29.1

So here are my questions:

-> Why my code is stuck in first row while there is no preselection in my "soup" variable?

-> Is there a way to make a loop that could check all the wanted elements for a different row each time? "data-row":"0" to "data-row":19 for instance.

Thanks for your support and have a great day !

Answer 1

It's stuck in the first row for a couple of reasons:

1. you are using the .find() which only returns the first element it "finds" in the html soup object.
2. You never iterate through anything. soup.find("td", {"data-stat": "avg_age"}).text will always return the same thing. Look at your loop.

Essentially this would be the same logic as you have there:

for x in [1, 2, 3, 4]:
    print(1)

As it iterates through that list, it's just going to print 1 and you will get the 1 4 times in your console.

You need to get all the rows in soup with soup.find_all('tr') . Then when you iterate, if there is a <td> class with attribute data-stat="avg_age" , only then do you want to .find() it and get the text.

import pandas as pd
from bs4 import BeautifulSoup
import requests


#Get the source code
r = requests.get('https://fbref.com/fr/comps/13/stats/Statistiques-Ligue-1#all_stats_standard')
c = r.content
soup = BeautifulSoup(c, "html.parser")
print(soup.prettify())


rows = soup.find_all('tr')
for a in rows:
    if a.find("td", {"data-stat" : "avg_age"}):
        print (a.find("td", {"data-stat" : "avg_age"}).text)

Output:

29.1
26.8
29.4
26.8
27.8
26.2
27.2
25.8
26.0
26.9
24.8
25.5
26.9
25.9
27.6
24.5
26.3
28.8
25.6
26.7
26.1
28.2
26.9
26.6
26.0
27.7
28.0
26.8
29.9
25.5
27.1
27.1
27.1
27.2
27.0
27.0
25.1
25.8
25.9
25.8

Just as note, pandas ' .read_html() uses bs4 under the hood to parse <table> tags. Use that. It's fair more easier.

import pandas as pd

df = pd.read_html('https://fbref.com/fr/comps/13/stats/Statistiques-Ligue-1#all_stats_standard', header=1)[0]

Output:

print(df)
           Équipe  # JC   Âge  Poss  MJ  ...  xG.1  xA.1  xG+xA  npxG.1  npxG+xA.1
0         Ajaccio    18  29.1  34.5   2  ...  0.59  0.14   0.73    0.20       0.34
1          Angers    18  26.8  55.0   2  ...  1.00  0.49   1.49    1.00       1.49
2         Auxerre    15  29.4  39.5   2  ...  0.43  0.43   0.85    0.43       0.85
3           Brest    18  26.8  42.5   2  ...  0.63  0.23   0.86    0.23       0.47
4   Clermont Foot    18  27.8  48.5   2  ...  0.17  0.07   0.24    0.17       0.24
5            Lens    16  26.2  63.0   2  ...  1.48  0.94   2.41    1.08       2.02
6           Lille    18  27.2  65.0   2  ...  2.02  1.65   3.66    2.02       3.66
7         Lorient    14  25.8  36.0   1  ...  0.37  0.26   0.63    0.37       0.63
8            Lyon    15  26.0  68.0   1  ...  1.52  0.49   2.00    0.73       1.22
9       Marseille    17  26.9  55.0   2  ...  1.10  0.89   1.99    1.10       1.99
10         Monaco    19  24.8  40.5   2  ...  2.75  1.21   3.96    2.36       3.57
11    Montpellier    19  25.5  47.5   2  ...  0.93  0.66   1.59    0.93       1.59
12         Nantes    16  26.9  40.5   2  ...  1.37  0.60   1.97    1.37       1.97
13           Nice    18  25.9  54.0   2  ...  0.49  0.40   0.88    0.49       0.88
14      Paris S-G    18  27.6  60.0   2  ...  3.05  1.76   4.81    2.27       4.03
15          Reims    18  24.5  43.0   2  ...  0.54  0.42   0.96    0.54       0.96
16         Rennes    17  26.3  65.0   2  ...  1.86  1.15   3.01    1.86       3.01
17     Strasbourg    18  28.8  49.5   2  ...  0.60  0.57   1.17    0.60       1.17
18       Toulouse    18  25.6  57.0   2  ...  0.58  0.58   1.15    0.58       1.15
19         Troyes    16  26.7  39.0   2  ...  0.91  0.23   1.14    0.52       0.75

[20 rows x 29 columns]

To print just the Age columns: print(df['Âge'])