Newspaper3k: Any way to download multiple web articles to one variable?

Question

I am trying to download a number of web articles for parsing. They are similar articles (annual reports), and I'd like all three to be downloaded in one singular output/variable for simplicity.

When I separate multiple urls, the code works, however, only the first url is downloaded.

Here is an example of the code I am trying to run:

from newspaper import Article

consolidated = Article('url1.com', 'url2.com', 'url3.com')
consolidate.download()
consolidated.parse()
consolidated.nlp()

result = consolidated.text
print(result)

url1

Answer 1

The constructor of Article is defined as follows :

class Article(object):
    """Article objects abstract an online news article page
    """
    def __init__(self, url, title='', source_url='', config=None, **kwargs):

So, if you do Article('url1.com', 'url2.com', 'url3.com') , that is interpreted by Python as

Article(url='url1.com', title='url2.com', source_url='url3.com')

which is not what you want.

Depending on your use, you could for example store all your articles in a dictionary like so:

from newspaper import Article
import nltk
nltk.download('punkt')

urls = ['https://edition.cnn.com/2021/03/24/americas/brazil-youth-covid-19-intl-latam/index.html', 
        'https://edition.cnn.com/2021/03/25/business/astrazeneca-covid-vaccine/index.html', 
        'https://edition.cnn.com/2021/03/25/india/india-covid-double-mutant-variant-intl-hnk/index.html']

results = {}

for url in urls:
    article = Article(url)
    article.download()
    article.parse()
    article.nlp()
    results[url] = article

You can then work with it later on like this:

for url in urls:
    print(url)
    article = results[url]
    print(article.authors)
    print(article.publish_date)
    print(article.keywords)
    print(article.summary)

Which outputs in this example:

https://edition.cnn.com/2021/03/24/americas/brazil-youth-covid-19-intl-latam/index.html
['Matt Rivers']
2021-03-24 00:00:00
['doctors', 'sick', 'silva', 'younger', 'icu', 'p1', 'patients', 'variant', 'getting', 'young', 'brazil', 'da', 'covid19']
São Paulo (CNN) It took only 10 days from start to finish, from the time 28-year-old Graciane da Silva got sick to the time she died.
She was alone when she passed away in a Rio de Janeiro Hospital -- her mom, Maria da Penha da Silva Siqueira, thinks about that often.
And amid that surge, a worrying pattern has emerged—more young people seem to be getting severely ill and dying from Covid-19, doctors tell CNN.
The increase in both illness and death in younger people has coincided with the rise of at least one Covid-19 variant in Brazil.
It's an effect that Maria da Penha da Silva Siqueira feels acutely nearly every day.
https://edition.cnn.com/2021/03/25/business/astrazeneca-covid-vaccine/index.html
['Julia Horowitz', 'Cnn Business', 'Andrew Berens', 'Svb Leerink Analyst', 'Ronn Torossian', 'Ceo Of Public Relations']
2021-03-25 00:00:00
['think', 'astrazenecas', 'mistake', 'trials', 'vaccine', 'data', 'doses', 'pandemic', 'european', 'astrazeneca', 'hero', 'went', 'villain', 'vaccines', 'company']
AstraZeneca updated its data on Thursday, reporting that the trials showed its vaccine to be 76% effective in preventing Covid-19 symptoms.
The AstraZeneca vaccine is administered to a patient at a pharmacy in London.
France also initially limited AstraZeneca vaccines to those under 65.
A nurse prepares a dose of the AstraZeneca vaccine at the Edouard Herriot hospital on Feb. 6 in Lyon, France.
Frustrations boiled over this week after 29 million doses of AstraZeneca's vaccine were discovered in a reported "raid" on a factory in Italy.
https://edition.cnn.com/2021/03/25/india/india-covid-double-mutant-variant-intl-hnk/index.html
['Jessie Yeung', 'Esha Mitra']
2021-03-25 00:00:00
['india', 'mutations', 'strain', 'variants', 'according', 'double', 'cases', 'vaccine', 'spike', 'variant', 'mutant', 'ministry', 'covid19', 'detects']
New Delhi (CNN) India has discovered a new "double mutant" variant of Covid-19 , as the country struggles to contain a spike in cases that's raising fears of a second wave.
A "double mutant" variant is a virus strain that carries two mutations.
India has now reported a total of more than 11.7 million cases and 160,000 related deaths, according to Johns Hopkins University data.
And now double mutations have been reported.
The Brazil variant known as P.1 has 17 mutations, and the South Africa variant known as B.1.351 also has multiple mutations, according to the US Centers for Disease Control and Prevention (CDC).