python 中的 requests.get(url) 在循环中使用时表现不同

Question

I'm new in python programming and trying to scrape every link available in my Urls.txt file. the code I wrote is:

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
user_agent = UserAgent()
fp = open("Urls.txt", "r")
values = fp.readlines()
fin = open("soup.html", "a")
for link in values:
    print( link )
    page = requests.get(link, headers={"user-agent": user_agent.chrome})
    html = page.content
    soup = BeautifulSoup(html, "html.parser")
    fin.write(str(soup))

The code works absolutely fine when the links are provided directly as string instead of as variable but when used as it is the output differs.

Answer 1

Maybe the string you read from the file has a line break. To remove it use link.strip("\n")