Python 将列表从文件传递到 requests.get()

Question

I'm trying to scrape a corpus of news article for analysis. I have a text file with a list of URL's and I'm trying to pass these to requests so that the page can be scraped with BeautifulSoup. I can pull the urls from the text file. However, I'm not properly passing that outputto requests.get(). When I give requests.get() an explicit url, the script works fine. How do I properly pass to requests.get() a list of links from a text file? Here is what I have working.

import requests
from bs4 import BeautifulSoup
r = requests.get("https://examplewebsite.org/page1")
coverpage = r.content
soup = BeautifulSoup(coverpage, 'html5lib')
file = open("output.txt", "w")
file.write("ITEM:")
paragraphs = soup.find_all("p")[11:-10]
for paragraph in paragraphs:
    file.write(paragraph.get_text())
    file.write("\n")
    file.write("\n")
file.close()

However, when I try to read from a text file that is a list of links, there seems to be a problem with how I'm passing the links to requests.get(). With one URL per line, the text file list of links looks like

https://examplewebsite.org/page1
https://examplewebsite.org/page2
https://examplewebsite.org/page3
https://examplewebsite.org/page4

Here is how I'm trying to work through the list of links.

f = open('article-list.txt', 'r')
urls = list(f)
for url in urls:
    import requests
    from bs4 import BeautifulSoup
    r = requests.get(url)
    coverpage = r.content
    soup = BeautifulSoup(coverpage, 'html5lib')
    file = open("output.txt", "w")
    file.write("ITEM:")
    paragraphs = soup.find_all("p")[11:-10]
    for paragraph in paragraphs:
        file.write(paragraph.get_text())
        file.write("\n")
        file.write("\n")
        print(paragraph.get_text())
file.close()

What I get is an error saying

AttributeError: 'NoneType' object has no attribute 'get_text'

This suggests to me I'm not properly passing the request. If I simply swap an explicitly defined url like " https://somewebsite.org/page1 " then the script works and writes paragraphs to the file. Yet when I put a print(urls) statement at the top and give requests.get() an explicit link so it does not break, I get a list of urls. However, that list is formatted as:

[' http://examplewebsite.org/page1 \n', ' http://examplewebsite.org/page2 \n', ' http://examplewebsite.org/page3 \n']

I think that \n is the problem. I tried running the links all together and that didn't work. Also for readability, I'd much prefer to have each link on a separate line. Any suggestions for how to address this would be deeply appreciated. Thanks.

Answer 1

In order to get the list just as they are in the file, this line

urls = list(f)

should look like this

urls = f.readlines()

It will return an array of every line in the txt file without any "\n"

Answer 2

Removing "\n" with the use of.rstrip() solved the issue. The code below is working and properly writes a group of news items to a single text file.

import requests
from bs4 import BeautifulSoup

f = open('article-list.txt', 'r')
urls_n = list(f)
urls = [url.rstrip("\n") for url in urls_n]

for url in urls:
    import requests
    from bs4 import BeautifulSoup
    r = requests.get(url)
    coverpage = r.content
    soup = BeautifulSoup(coverpage, 'html5lib')
    file = open("output.txt", "a")
    file.write("ITEM:")
    paragraphs = soup.find_all("p")[11:-10]
    for paragraph in paragraphs:
        file.write(paragraph.get_text())
        file.write("\n")
        file.write("\n")
        print(paragraph.get_text())
file.close()