How do I loop through pages request?

Question

I am very (very) new to python and am struggling to get my loop to go through pages in the request-it seems only to be returning the first page of results so I can only think that I have missed a vital part of code.....here is what I have so far:

import requests

articles = []
for i in range(1, 6):
    response = requests.get(url=everything_news_url, headers=headers, params=everything_payload)
    headers = {'Authorization': 'xxxxxxxxxxxxxxxxxxxx'}
    everything_news_url = 'https://newsapi.org/v2/everything'
    everything_payload = {
        'q': 'cryptocurrency',
        'language': 'en',
        'sortBy': 'relevancy',
        'from_param': '2019-10-20',
        'to': '2019-11-11',
        'page': 'i'
    }
    headlines_payload = {'category': 'business', 'country': 'us'}
    sources_payload = {'category': 'general', 'country': 'us'}

articles.append(response)

Any help greatly appreciated...no errors show but only the first page of results!

Answer 1

You had forgot to ident the code into the for, and also you had your usage of i as a string, also, some of it didn't need to be inside the loop.

headers = {'Authorization': 'xxxxxxxxxxxxxxxxxxxx'}
everything_news_url = 'https://newsapi.org/v2/everything'
headlines_payload = {'category': 'business', 'country': 'us'}
sources_payload = {'category': 'general', 'country': 'us'}
articles = []
for i in range (1, 6):              
  everything_payload = {'q': 'cryptocurrency', 'language': 'en', 'sortBy': 'relevancy',
  'from_param' : '2019-10-20', 'to':'2019-11-11', 'page': i }
  response = requests.get(url=everything_news_url,
                          headers=headers,
                          params=everything_payload)
articles.append(response)

Answer 2

I don't know more than the basics of python, but it seems to me that it's a simple syntax error. Try the following:

articles = []
for i in range (1, 6):    
    response = requests.get(url=everything_news_url, 
    headers=headers,params=everything_payload)
    headers = {'Authorization': 'xxxxxxxxxxxxxxxxxxxx'}
    everything_news_url = 'https://newsapi.org/v2/everything'
    everything_payload = {'q': 'cryptocurrency', 'language': 'en', 'sortBy': 'relevancy', 
    'from_param' : '2019-10-20', 'to':'2019-11-11', 'page': 'i'}
    headlines_payload = {'category': 'business', 'country': 'us'}
    sources_payload = {'category': 'general', 'country': 'us'}
    articles.append(response)

Basically it seems like in the code you posted, you were only appending to articles one time as none of your code was actually in the for-loop.

Answer 3

First of all your code is not indented properly and define those variables before using them in requests's parameter.

import requests

articles = []
for i in range (1, 6):    

    headers = {'Authorization': 'you api key'}
    everything_news_url = 'https://newsapi.org/v2/everything'
    everything_payload = {'q': 'cryptocurrency', 'language': 'en', 'sortBy': 'relevancy', 
    'from_param' : '2019-10-20', 'to':'2019-11-11', 'page': 'i'}
    headlines_payload = {'category': 'business', 'country': 'us'}
    sources_payload = {'category': 'general', 'country': 'us'}
    response = requests.get(url=everything_news_url, 
    headers=headers,params=everything_payload)
    articles.append(response)

print(articles)

Answer 4

The code had improper indentation and i was being used as string. The variables being passed as parameters for requests.get are being defined after the call. The functionality of the for loop is not being served as the append statement is outside the body of the loop. Assigning the same values to variables repeatedly in the loop is not elegant either.

articles = []
headers = {'Authorization': 'xxxxxxxxxxxxxxxxxxxx'}
everything_news_url = 'https://newsapi.org/v2/everything'
headlines_payload = {'category': 'business', 'country': 'us'}
sources_payload = {'category': 'general', 'country': 'us'}

for i in range (1, 6):    
    everything_payload = {'q': 'cryptocurrency', 'language': 'en', 'sortBy': 'relevancy', 'from_param' : '2019-10-20', 'to':'2019-11-11', 'page': i}
    response = requests.get(url=everything_news_url, headers=headers,params=everything_payload)
    articles.append(response)