Python requests - using twitter search

Question

I am trying to use requests to get data from twitter but when i run my code i get this error: simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is my code so far:

import requests

url = 'https://twitter.com/search?q=memes&src=typed_query'
results = requests.get(url)
better_results = results.json()
better_results['results'][1]['text'].encode('utf-8')
print(better_results)

Answer 1

because you are making a request to a dynamic website.

when we are making a request to a dynamic website we must render the html first in order to receive all the content that we were expecting to receive.

just making the request is not enough.

other libraries such as requests_html render the html and javascript in background using a lite browser .

you can try this code:

# pip install requests_html
from requests_html import HTMLSession

url = 'https://twitter.com/search?q=memes&src=typed_query'

session = HTMLSession()
response = session.get(url)

# rendering part
response.html.render(timeout=20)

better_results = response.json()
better_results['results'][1]['text'].encode('utf-8')
print(better_results)