使用 Python ZC49DFB55F06BB4576E2C5CA78DZ 文件循环通过 CSV 文件中的 URL 链接

Question

I've got a collection of URL's in a csv file and I want to loop through these links and open each link in the CSV one at a time. I'm getting several different errors depending on what I try but nonetheless I can't get the browser to open the links. The print shows that the links are there.

When I run my code i get the following error:

Traceback (most recent call last):
  File "/Users/Main/PycharmProjects/ScrapingBot/classpassgiit.py", line 26, in <module>
    open = browser.get(link_loop)
TypeError: Object of type bytes is not JSON serializable 

Can someone help me with my code below if I am missing something or if i am doing it wrong.

My code:

import csv
from selenium import webdriver
from bs4 import BeautifulSoup as soup
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as browser_wait
from selenium.webdriver.support import expected_conditions as EC
import requests

browser = webdriver.Chrome(executable_path=r'./chromedriver')

contents = []

with open('ClassPasslite.csv', 'rt') as cp_csv:
    cp_url = csv.reader(cp_csv)
    for row in cp_url:
        links = row[0]
        contents.append(links)

for link in contents:
    url_html = requests.get(links)

    for link_loop in url_html:

        print(contents)

        open = browser.get(link_loop)

Answer 1

Apparently, you are messing something up with the names. Without having a copy of the.csv file, I cannot reproduce the error - hence, I will assume that you correctly extract the link from the text file.

In the second part of your code, you use requests.get to get the links (mind the plural) option, but links apparently is an element that you define in the previous section ( links = row[0] ), whereas link is the actual object you define in the for loop. Below you can find a version of the code that might be a helpful starting point.

Let me add, though, that the contemporaneous use of requests and selenium in this case makes little sense in your context: why getting an HTML page and then loop over its elements to get other pages with selenium ?

import csv
import requests

browser = webdriver.Chrome(executable_path=r'./chromedriver')

contents = []

with open('ClassPasslite.csv', 'rt') as cp_csv:
    cp_url = csv.reader(cp_csv)
    for row in cp_url:
        links = row[0]
        contents.append(links)

for link in contents:
    url_html = requests.get(link) # now this is singular

    # Do what you have to do here with requests, in spite of using selenium #

Answer 2

Since you have not provided any form of what is contained in your variable contents I will assume that it is a list of url strings.

As @cap.py mentioned you are messing up by using requests and selenium at the same time. When you do a GET web request, the server at the destination will send you a text response. This text can be simply some text, like Hello world! or it can be some html. But this html code as to be interpreted in your computer which sent the request.

That's the point of selenium over requests: requests return the text gathered from the destination (url) while selenium ask a browser (eg Chrome) to do gather the text and if this text is some html, to interpret it to give you a real readable web page. Moreover the browser is running the javascript inside your page so dynamic pages works as well.

In the end the only thing needed to run your code is to do this:

import csv
from selenium import webdriver
from bs4 import BeautifulSoup as soup
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as browser_wait
from selenium.webdriver.support import expected_conditions as EC
import requests

browser = webdriver.Chrome(executable_path=r'./chromedriver')

contents = []

with open('ClassPasslite.csv', 'rt') as cp_csv:
    cp_url = csv.reader(cp_csv)
    for row in cp_url:
        links = row[0]
        contents.append(links)

#link should be something like "https://www.classpass.com/studios/forever-body-coaching-london?search-id=49534025882004019"
for link in contents:
    browser.get(link)
    # paste the code you have here

Tip: Don't forget that browsers take some time to load pages. Adding some time.sleep(3) will help you a lot.