JavaScript Disabled error while web scraping twitter in Python in BeautifulSoup

Question

I am new to this world of web scraping . I was trying to scrape twitter with BeautifulSoup in Python.

Here's my code:

from bs4 import BeautifulSoup
import requests

request = requests.get("https://twitter.com/mybmc").text

soup = BeautifulSoup(request, 'html.parser')

print(soup.prettify())

But I am getting a large output which is not the twitter page which I am looking for but there is a error container: Output Image

which says JavaScript is disabled in this browser. I tried changing my default browsers to Chrome, Firefox and Microsoft Edge but the out was same.

What should I do in this case?

Answer 1

Twitter here seem to be specifically trying to prevent scrapers of the front end, probably with the view that you should use their REST API to fetch that same data. It is not to do with your default browsers, but that requests.get will be providing a python requests user agent, which specifically doesn't support Javascript.

I'd suggest using a different page to practice on, or if it must be the twitter front page, consider using selenium perhaps with a standalone container to scrape.