使用dataframe和xlsxwriter内部的for循环将整个Beautifulsoup数组保存到excel中

Question

After going through a lot of documentation and looking for stackoverflow for answers I just couldn't find a solution to my issue.

Basically I am using beautifulsoup to scrape a list of data from a website and then store it into excel. Scraping works fine.

When I run my script it will print out all of the items to the terminal. However when I try to save this result into dataframe and save it to Excel it will only execute last line only and saves that one to excel.

I've tried storing the code inside the loop but same result. I've tried converting the list back into array inside for loop but same issue. Still last line only gets saved into Excel

I think I am missing a logical approach here. If someone could link me what to look for I would appreciate it a lot.

        soup = BeautifulSoup(html, features="lxml")
        soup.find_all("div", {"id":"tbl-lock"})

        for listing in soup.find_all('tr'):

            listing.attrs = {}

            assetTime = listing.find_all("td", {"class": "locked"})
            assetCell = listing.find_all("td", {"class": "assetCell"})
            assetValue = listing.find_all("td", {"class": "assetValue"})

            for data in assetCell:

                array = [data.get_text()]

                ### Excel Heading + data
                df = pd.DataFrame({'Cell': array
                                    })
               print(array)
                # In here it will print all of the data


        ### Now we need to save the data to excel
        ### Create a Pandas Excel writer using XlsxWriter as the Engine
        writer = pd.ExcelWriter(filename+'.xlsx', engine='xlsxwriter')

        ### Convert the dataframe to an XlsxWriter Excel object and skip first row for custom header
        df.to_excel(writer, sheet_name='SheetName', startrow=1, header=False)

        ### Get the xlsxwritert workbook and worksheet objects

        workbook = writer.book
        worksheet = writer.sheets['SheetName']

        ### Custom header for Excel
        header_format = workbook.add_format({
            'bold': True,
            'text_wrap': True,
            'valign': 'top',
            'fg_color': '#D7E4BC',
            'border': 1
        })

        ### Write the column headers with the defined add_format
        print(df) ### In here it will print only 1 line
        for col_num, value in enumerate(df):

            worksheet.write(0, col_num +1, value, header_format)

            ### Close Pandas Excel writer and output the Excel file
            writer.save()

Answer 1

This line is the problem df = pd.DataFrame({'Cell': array}) Here you're overwriting df, hence only storing the last line.

Instead, initialize df as df = pd.DataFrame(columns=['cell']) and in the loop do this

df = df.append(pd.DataFrame({'Cell': array}),ignore_index=True)

EDIT :

Try this

soup = BeautifulSoup(html, features="lxml")
soup.find_all("div", {"id":"tbl-lock"})

df = pd.DataFrame(columns=['cell'])
for listing in soup.find_all('tr'):

        listing.attrs = {}

        assetTime = listing.find_all("td", {"class": "locked"})
        assetCell = listing.find_all("td", {"class": "assetCell"})
        assetValue = listing.find_all("td", {"class": "assetValue"})

        for data in assetCell:

            array = [data.get_text()]

            ### Excel Heading + data
            df = df.append(pd.DataFrame({'Cell': array}),ignore_index=True)
            ##Or this
            #df = df.append(pd.DataFrame({'Cell': array}))   

            print(array)
            # In here it will print all of the data

. . . . Rest of Code