Python BeautifulSoup and Pandas extract table from list of urls and save all the tables into single dataframe or save as csv

Question

I am trying to extract tabular data from a list of urls and I want to save all the table into a single csv file.

I am new and relatively beginner in python and from non-CS background, however I am very eager to learn the same.

import pandas as pd
import urllib.request
import bs4 as bs

urls = ['A', 'B','C','D',...'Z']

for url in urls:
    source = urllib.request.urlopen(url).read()
    soup = bs.BeautifulSoup(source,'lxml')
    table = soup.find('table', class_='tbldata14 bdrtpg')
    table_rows = table.find_all('tr')

data = []
for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.text for tr in td]
    data.append(row)

final_table = pd.DataFrame(data, columns=["ABC", "XYZ",...])
final_table.to_csv (r'F:\Projects\McData.csv', index = False, header=True)

What I Get from above code in newly created csv file is -

ABC XYZ PQR MNL CYP ZXS
1   2   3   4   5   6

My above code only gets table from last url- 'Z' , which, as I have checked is actually the table from last url in list.

What I am trying to achieve here is getting all tables from list of urls - ie A to Z into single csv file.

Answer 1

This is an issue with indentation and order. table_rows gets reset every time through the for url in urls loop, so you only end up with the last URLs worth of data. If you want all of the URLs worth of data in one final CSV, see the changes I made below.

import pandas as pd
import urllib.request
import bs4 as bs

urls = ['A', 'B','C','D',...'Z']
data = [] # Moved to the start
for url in urls:
    source = urllib.request.urlopen(url).read()
    soup = bs.BeautifulSoup(source,'lxml')
    table = soup.find('table', class_='tbldata14 bdrtpg')
    table_rows = table.find_all('tr')

    #indented the following loop so it runs with every URL data
    for tr in table_rows:
        td = tr.find_all('td')
        row = [tr.text for tr in td]
        data.append(row)

final_table = pd.DataFrame(data, columns=["ABC", "XYZ",...])
final_table.to_csv (r'F:\Projects\McData.csv', index = False, header=True)