How to find elements that match specific conditions in Beautiful Soup

Question

I am learning and trying both Python (2.7) and Beautiful Soup (3.2.0) . I already got some help here with my first problems ( Beautiful Soup throws `IndexError` )

This is the Python code so far:

# Import the classes that are needed
import urllib2
from BeautifulSoup import BeautifulSoup

# URL to scrape and open it with the urllib2
url = 'http://www.wiziwig.tv/competition.php?competitionid=92&part=sports&discipline=football'
source = urllib2.urlopen(url)

# Turn the saced source into a BeautifulSoup object
soup = BeautifulSoup(source)

# From the source HTML page, search and store all <div class="date">...</div> and it's content
datesDiv = soup.findAll('div', { "class" : "date" })
# Loop through the tag and store only the needed information, being the actual date
dates = [tag.contents[0] for tag in datesDiv]

# From the source HTML page, search and store all <span class="time">...</span> and it's content
timesSpan = soup.findAll('span', { "class" : "time" })
# Loop through the tag and store only the needed information, being the actual times
times = [tag.contents[0] for tag in timesSpan]

# From the source HTML page, search and store all <td class="home">..</td> and it's content
hometeamsTd = soup.findAll('td', { "class" : "home" })
# Loop through the tag and store only the needed information, being the home team
# if tag.contents[1] != 'Dutch KNVB Beker' - Do a direct test if output is needed or not
hometeams = [tag.contents[1] for tag in hometeamsTd if tag.contents[1] != 'Dutch KNVB Beker']

# From the source HTML page, search and store all <td class="away">..</td> and it's content
# [1:] at the end meand slice the first one found
awayteamsTd = soup.findAll('td', { "class" : "away" })[1:]
# Loop through the tag and store only the needed information, being the away team
awayteams = [tag.contents[1] for tag in awayteamsTd]

# From the source HTML page, search and store all <a class="broadcast" href="...">..</a> and it's content
broadcastsA = soup.findAll('a', { "class" : "broadcast" })
# Loop through the tag and store only the needed information, being the the broadcast URL, where we can find the streams
broadcasts = [tag['href'] for tag in broadcastsA]

The problem I got is that the arrays are not equal to each other:

len(dates)      #9, should be 6
len(times)      #18, should be 12
len(hometeams)  #6, is correct
len(awayteams)  #6, is correct
len(broadcasts) #9, should be 6

Problem I have is that I do the following search for getting the dates array: soup.findAll('div', { "class" : "date" }) . Which obviously gives me all the <div> elements with class="date" . But the problem is, that I only need the date when there is also a <td> element with class="away" .

See next part of the HTML that I am scraping:

<tr class="odd">
    <td class="logo">
        <img src="/gfx/disciplines/football.gif" alt="football"/>
    </td>
    <td>
        <a href="/competition.php?part=sports&amp;competitionid=92&amp;discipline=football">Dutch Cup</a>
        <img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="comp-92"/>
    </td>
    <td>
        <div class="date" rel="1380054900">Tuesday, September 24</div> <!-- This date is not needed, because within this <tr> there is no <td class="away"> -->
        <span class="time" rel="1380054900">22:35</span> - <!-- This time is not needed, because within this <tr> there is no <td class="away"> -->
    <span class="time" rel="1380058500">23:35</span> <!-- This time is not needed, because within this <tr> there is no <td class="away"> -->
    </td>
    <td class="home" colspan="3">
        <img class="flag" src="/gfx/flags/nl.gif" alt="nl"/>Dutch KNVB Beker<img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-6758"/>
    </td>
    <td class="broadcast">
        <a class="broadcast" href="/broadcast.php?matchid=221554&amp;part=sports">Live</a> <!-- This href is not needed, because within this <tr> there is no <td class="away"> -->
    </td>
</tr>
<tr class="even">
    <td class="logo">
        <img src="/gfx/disciplines/football.gif" alt="football"/>
    </td>
    <td>
        <a href="/competition.php?part=sports&amp;competitionid=92&amp;discipline=football">Dutch Cup</a>
        <img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="comp-92"/>
    </td>
    <td>
        <div class="date" rel="1380127500">Wednesday, September 25</div> <!-- This date we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
        <span class="time" rel="1380127500">18:45</span> - <!-- This time we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
    <span class="time" rel="1380134700">20:45</span> <!-- This date we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
    </td>
    <td class="home">
        <img class="flag" src="/gfx/flags/nl.gif" alt="nl"/>PSV<img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-3"/>
    </td>
    <td>vs.</td>
    <td class="away">
        <img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-428"/>Stormvogels Telstar<img class="flag" src="/gfx/flags/nl.gif" alt="nl"/>
    </td>
    <td class="broadcast">
        <a class="broadcast" href="/broadcast.php?matchid=221555&amp;part=sports">Live</a> <!-- This href we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
    </td>
</tr>

Answer 1

How about rethinking the way you scrape the data. You have a table with matches - then just iterate over the rows:

for tr in soup.findAll('tr', {'class': ['odd', 'even']}):
    home_team = tr.find('td', {'class': 'home'}).text
    if home_team == 'Dutch KNVB Beker':
        continue

    away_team = tr.find('td', {'class': 'away'}).text
    date = ' - '.join([span.text for span in tr.findAll('span', {'class': 'time'})])
    broadcast = tr.find('a', {'class': 'broadcast'})['href']

    print home_team, away_team, date, broadcast

prints 5 rows:

RKC Waalwijk Heracles 20:45 - 22:45 /broadcast.php?matchid=221553&part=sports
PSV Stormvogels Telstar 18:45 - 20:45 /broadcast.php?matchid=221555&part=sports
Ajax FC Volendam 20:45 - 22:45 /broadcast.php?matchid=221556&part=sports
SC Heerenveen FC Twente 18:45 - 20:45 /broadcast.php?matchid=221558&part=sports
Feyenoord FC Dordrecht 20:45 - 22:45 /broadcast.php?matchid=221559&part=sports

Then, you can collect results into the list of dicts.