如何修复熊猫中的“无法设置列不匹配的行”错误

Question

I'm creating a web scraper for a project of mine. I'm web scraping jobs from indeed. I'm able to get all the data that I need. Now I'm having a problem creating a dataframe to save it to a CSV file.

I have searched for the error and tried many possible solutions but I keep getting the same error. Appreciate any suggestions on code or error problem. Thank you

ValueError: cannot set a row with mismatched columns

import requests
import bs4
from bs4 import BeautifulSoup

import pandas as pd
import time


max_results_per_city = 30

city_set = ['New+York','Chicago']
columns = ["city", "job_title", "company_name", "location", "summary"]

database = pd.DataFrame(columns = columns)

for city in city_set:
    for start in range(0, max_results_per_city, 10):
        page = requests.get('https://www.indeed.com/jobs?q=computer+science&l=' + str(city) + '&start=' + str(start))
        time.sleep(1)
        soup = BeautifulSoup(page.text, "lxml")
        for div in soup.find_all(name="div", attrs={"class":"row"}):
            num = (len(sample_df) + 1)
            job_post = []
            job_post.append(city)
            for a in div.find_all(name="a", attrs={"data-tn-element":"jobTitle"}):
                job_post.append(a["title"])
            company = div.find_all(name="span", attrs={"class":"company"})
            if len(company) > 0:
                for b in company:
                    job_post.append(b.text.strip())
            else:
                sec_try = div.find_all(name="span", attrs={"class":"result-link-source"})
                for span in sec_try:
                    job_post.append(span.text)
            
            c = div.findAll('div', attrs={'class': 'location'})
            for span in c:
                 job_post.append(span.text)
            d = div.findAll('div', attrs={'class': 'summary'})
            for span in d:
                job_post.append(span.text.strip())
            database.loc[num] = job_post
            database.to_csv("test.csv")

Answer 1

This issue is caused by the # Columns not matching the amount of data (for at least one row).

I see a number of issues: Where is 'sample_df' initialized, where are you adding data to 'database' are the big ones that pop out.

I'd restructure your code job_post looks like your row level list. I would use to append to a table level list, so at the end of each loop hit table.append(job_post) instead of sample_df.loc[num] = job_post

then after your loop you can call Dataframe(table, columns=columns)

a note: make sure you're adding None, Null or "" when your scraper can't find data, otherwise your row length wont match your column length, which is what is causing your error.

Answer 2

Reproducing your code, it was not extracting location and database indentation is in the wrong place. So, fix c = div.findAll(name='span', attrs={'class': 'location'}) . Here's a fix that makes it work:

database = []

for city in city_set:
    for start in range(0, max_results_per_city, 10):
        page = requests.get('https://www.indeed.com/jobs?q=computer+science&l=' + str(city) + '&start=' + str(start))
        time.sleep(1)
        soup = BeautifulSoup(page.text, "lxml")
        for div in soup.find_all(name="div", attrs={"class":"row"}):
            #num = (len(sample_df) + 1)
            job_post = []
            job_post.append(city)
            for a in div.find_all(name="a", attrs={"data-tn-element":"jobTitle"}):
                job_post.append(a["title"])
            company = div.find_all(name="span", attrs={"class":"company"})
            if len(company) > 0:
                for b in company:
                    job_post.append(b.text.strip())
            else:
                sec_try = div.find_all(name="span", attrs={"class":"result-link-source"})
                for span in sec_try:
                    job_post.append(span.text)

            c = div.findAll(name='span', attrs={'class': 'location'})
            for span in c:
                 job_post.append(span.text)
            d = div.findAll('div', attrs={'class': 'summary'})
            for span in d:
                job_post.append(span.text.strip())
        database.append(job_post)

df00=pd.DataFrame(database)
df00.shape


df00.columns=columns
df00.to_csv("test.csv",index=False)