Extracting data from a web page using BS4 in Python

Question

I am trying to extract data from this site: http://www.afl.com.au/fixture

in a way such that I have a dictionary having the date as key and the "Preview" links as Values in a list, like

dict = {Saturday, June 07: ["preview url-1, "preview url-2","preview url-3","preview url-4"]}

Please help me get it, I have used the code below:

def extractData():
    lDateInfoMatchCase = False
#     lDateInfoMatchCase = []
    global gDict
    for row in table_for_players.findAll("tr"):
        for lDateRowIndex in row.findAll("th", {"colspan" : "4"}):
            ldateList.append(lDateRowIndex.text)

    print ldateList
    for index in ldateList:
        #print index
        lPreviewLinkList = []
        for row in table_for_players.findAll("tr"):
            for lDateRowIndex in row.findAll("th", {"colspan" : "4"}):

                if lDateRowIndex.text == index:
                    lDateInfoMatchCase = True
                else:
                    lDateInfoMatchCase = False

             if lDateInfoMatchCase == True:
                     for lInfoRowIndex in row.findAll("td", {"class": "info"}):
                         for link in lInfoRowIndex.findAll("a", {"class" : "preview"}):
                             lPreviewLinkList.append("http://www.afl.com.au/" + link.get('href'))
        print lPreviewLinkList
        gDict[index] = lPreviewLinkList

My main aim is to get the all player names who are playing for a match in home and in away team according to date in a data structure.

Answer 1

I prefer using CSS Selectors . Select the first table, then all rows in the tbody for ease of processing; the rows are 'grouped' by tr th rows. From there you can select all next siblings that don't contain th headers and scan these for preview links:

previews = {}

table = soup.select('table.fixture')[0]
for group_header in table.select('tbody tr th'):
    date = group_header.string
    for next_sibling in group_header.parent.find_next_siblings('tr'):
        if next_sibling.th:
            # found a next group, end scan
            break
        for preview in next_sibling.select('a.preview'):
            previews.setdefault(date, []).append(
                "http://www.afl.com.au" + preview.get('href'))

This builds a dictionary of lists; for the current version of the page this produces:

{u'Monday, June 09': ['http://www.afl.com.au/match-centre/2014/12/melb-v-coll'],
 u'Sunday, June 08': ['http://www.afl.com.au/match-centre/2014/12/gcfc-v-syd',
                      'http://www.afl.com.au/match-centre/2014/12/fre-v-adel',
                      'http://www.afl.com.au/match-centre/2014/12/nmfc-v-rich']}