在 for 循环中创建新的变量/类实例？ Python web刮

Question

I am currently working on a web scraper that will take urls as inputs, find the page, scrape it, then return results in a CSV. The scraper works well for single URL's at a time. But unfortunately whenever it writes a new line to the scrape results CSV it also appends the previous url's scrape results in each column. I need a loop that will essentially create new class variables inside the loop so that this doesn't happen. Something like that does this: Takes list of urls, then also creates unique class instance.

links = ['www.SomeLink1.com','www.Somelink2.com','www.SomeLink3.com']


person1 = Person('www.SomeLink1.com', driver = driver, close_on_complete = False)
person2 = Person('www.Somelink2.com', driver = driver, close_on_complete = False)
person3 = Person('www.SomeLink3.com', driver = driver, close_on_complete = False) 

I do not have access to the source code to create a new method "person1.reset()" or something.

Here is also the original code I was using to scrape multiple pages:

# Import libraries
from linkedin_scraper import Person, actions
from selenium import webdriver
import csv
import os
import pandas as pd
import numpy as np
import smtplib

# Read-in list of contacts:
contacts = pd.read_csv("/Users/Desktop/ContactList.csv")
names = contacts['contact_name'].tolist()
urls = contacts['contact_url'].tolist()
# turn contacts list into dictionary just in case
contact_dict = {names[i]: urls[i] for i in range(len(names))}
print(contact_dict)

# automatically login to LinkedIn
driver = webdriver.Chrome('/Users/Downloads/chromedriver')
email = os.environ.get('LINKEDIN_USER')
password = os.environ.get('LINKEDIN_PASS')
actions.login(driver, email, password)

# create general field names
fields = ['name', 'about', 'job_title', 'location','company',
          'education','accomplishments','linkedin_url']

with open('ScrapeResults.csv', 'w') as f:
    # using csv.writer method from CSV package
    write = csv.writer(f)
    write.writerow(fields)
f.close()

# Loop-through urls to scrape multiple pages at once
for individual,link in contact_dict.items():

    ## assign ##
    the_name = individual
    the_link = link
    # scrape peoples url:
    person = Person(the_link, driver=driver, close_on_complete=False)

    # rows to be written... only index for lists?
    rows = [[person.name, person.about, person.job_title, person.location, person.company,
             person.educations, person.accomplishments, person.linkedin_url]]
    # write
    with open('ScrapeResults.csv', 'a') as f:
    # using csv.writer method from CSV package
        write = csv.writer(f)
        write.writerows(rows)
        f.close()

Answer 1

Could you try instantiating a new driver each time? That should reset counters in driver for you.

for individual,link in contact_dict.items():
    the_name = individual
    the_link = link
    driver = Driver() # I don't know how to instantiate this
    person = Person(the_link, driver=driver, close_on_complete=False).

Without access to driver documentation, I cannot speak to how to properly instantiate it. As well, it might even have a helper to clear() or reset() internal variables which would be preferable to recreating the driver from scratch. In any case, the scraper should have straightforward documentation for this.

Answer 2

Got in touch with creator of "linkedin_scraper" library. He fixed a bug that cached previous linkedin profile values/accumulated them when scraping multiple at once.

Issue resolved in version 2.7.5.

Please see: https://github.com/joeyism/linkedin_scraper/issues/84

Thanks all!