Web scraping the table from the website

Question

Hi I am trying to grab and parse all table data from the web https://html5test.com/ . so, I wrote the following code.But its not showing any data. I looked over the question answer but can't figure out the problem.

from BeautifulSoup import BeautifulSoup
from urllib2 import urlopen
import re


url='https://html5test.com/'
data=urlopen(url)

parse=BeautifulSoup(data).findAll('div', attrs={'class': 'resultsTable detailsTable'})

Answer 1

Viewing the source code (view-source: https://html5test.com/ in Chrome), I don't actually find the 'resultsTable` class. It looks like this is generated dynamically using JavaScript. You need a scraper which renders JavaScript, for example, Scrapy with Splash (cf. https://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash/ ).