用BeautifulSoup提取表数据

Question

I would like to get table data from the website below:

https://www.isyatirim.com.tr/tr-tr/analiz/hisse/Sayfalar/sirket-karti.aspx?hisse=ALCAR

My code gets table data but I am experiencing a problem appending to dataframe.

stock = 'ALCAR'
page = requests.get(f"https://www.isyatirim.com.tr/tr-tr/analiz/hisse/Sayfalar/sirket-karti.aspx?hisse={stock}")

soup = BeautifulSoup(page.content, 'html.parser')

table = soup.find('tbody', id="tbodyMTablo")
print(table)


for j in table.find_all('tr'):
    row_data = j.find_all('td')
    row = [i.text for i in row_data]

    #print(row)
    df = pd.DataFrame(row).transpose()
    df.to_csv('xxx.csv')
    print(df)

Answer 1

You are saving each row in a csv file overwriting it at each step in the for loop. Here's a solution that allows to first save all rows in a pandas dataframe and finally save the dataframe to csv.

import requests
from bs4 import BeautifulSoup
import pandas as pd

stock = 'ALCAR'
page = requests.get(f"https://www.isyatirim.com.tr/tr-tr/analiz/hisse/Sayfalar/sirket-karti.aspx?hisse={stock}")

soup = BeautifulSoup(page.content, 'html.parser')

table = soup.find('tbody', id="tbodyMTablo")

df = pd.DataFrame() # initialize dataframe
for j in table.find_all('tr'):
    row_data = j.find_all('td')
    row = [i.text for i in row_data]
    df = df.append(pd.DataFrame(data=[row])) # append row to dataframe

df.to_csv('xxx.csv')  # save dataframe to csv file
print(df)

Answer 2

Just in addition - Cause you are working with pandas you can simply use pandas.read_html to create your DataFrame .

import pandas as pd

stock = 'ALCAR'
df = pd.read_html(f'https://www.isyatirim.com.tr/tr-tr/analiz/hisse/Sayfalar/sirket-karti.aspx?hisse={stock}', attrs = {'class':'excelexport'})[0]

df.to_csv('filename.csv')

Or if you do not need headers and index :

df.to_csv('filname.csv', header=None,index=None)