I am getting an unexpected output while exporting my dictionary to csv file in python

Question

I was taking data from a table through beautiful-soup and want to paste it's output in a csv file. Suppose i want data from 1 to 10, But my csv file shows the last value only, ie 9th. And my header is also typed repeatedly in every output. Please see my code and show me where i am wrong. However my code has no errors , but it is producing data in a strange manner in terminal and printing just only the last value of data in csv file. : (

from bs4 import BeautifulSoup
import requests
from pandas import DataFrame


max = 100500
i = 100002
while i < max:
    page = requests.get(
        "https://rajasthanpuc.in/P_test.aspx?Spuc=P379RJ14" + str(i) + "&flag=1")
    i = i + 1

    soup = BeautifulSoup(page.content, 'html.parser')

    num = soup.find(id='Cmobile_lbl').get_text()
    num2 = soup.find(id='Vechno_lbl').get_text()
    num3 = soup.find(id='Pucc_lbl').get_text()
    num4 = soup.find(id='customername_lbl').get_text()
    num5 = soup.find(id='Regyear_lbl').get_text()
    num6 = soup.find(id='vehtype_lbl').get_text()
    num7 = soup.find(id='EngType_lbl').get_text()
    num8 = soup.find(id='Make_lbl').get_text()
    num9 = soup.find(id='Model_lbl').get_text()
    num10 = soup.find(id='Fuel_lbl').get_text()
    num11 = soup.find(id='TestDate_lbl').get_text()
    num12 = soup.find(id='TestTime_lbl').get_text()
    num13 = soup.find(id='validdate_lbl').get_text()
    num16 = soup.find(id='licence_lbl').get_text()
    num17 = soup.find(id='txtResult').get_text()


# print(text)

    df = DataFrame({
        'numbers______': [num],
        'vehicle_no___': [num2],
        'fuel___': [num10],
        'Pucc___': [num3],
        'regn_________': [num5],
        'model________________': [num9],
        'name_________________': [num4]
    })

    print(df)

    df.to_csv(r'C:\Users\intel\Desktop\export_dataframe.csv',
              index=False, encoding='utf-8')

Answer 1

Move print(df) outside the loop. This will prevent printing the header over and over.

Finally, use the different file mode when writing the csv to file. According to the docs , the df.to_csv(...) method can accept a mode parameter and defaults to w .

mode : str

• Python write mode, default 'w'.

So change the mode to something else like a which will open the file for appending without first truncating it.

df.to_csv(r'C:\Users\intel\Desktop\export_dataframe.csv',
          mode='a', index=False, encoding='utf-8')