Scraping HTML data with JavaScript or Python

Question

I would like to scraping data from html and save it to text file I have the URL can you please If JavaScript will be preferable or python I can try

# Import libraries
import requests
import urllib2
import time from bs4
import BeautifulSoup

# Set the URL you want to webscrape from
url = 'theweathernetwork.com/ca/hourly-weather-forecast/ontario/london'

# Connect to the URL
response = requests.get(url)

# Parse HTML and save to BeautifulSoup object
soup = BeautifulSoup(response.text, "html.parser")

# To download the whole data set, let's do a for loop through all a tags

Answer 1

You can get the data from the api (found through looking at DevTools -> XHR). But if you're trying to get the html, you'll need to use something like selenium so the page can render, and THEN grab the html source.

So, not sure if this is what you want, but the data is there and you can pull what ever it is you want to pull. Here is an example of the hourly data.

import requests
import pandas as pd

url = 'https://www.theweathernetwork.com/api/data/caon0383/hourly/cm/ci?ts=1012'

jsonData = requests.get(url).json()

df = pd.DataFrame(jsonData['hourly']['periods'])

Output:

print (df.to_string())
                   b cc_class                  cdate  cloud_coverage dayname_alt dewpt_unit   dn   f  fc feelsLikeNight_unit fu   hour       ic icon              ii                        it   ms   n pop_class  pp      r  rain_bar_height rain_unit_language rain_value  rr  ru      s         sd  showrainunit  showsnowunit sky_tenths  snow_bar_height snow_unit_language snow_value  sr  su   t  tc tmau tmu            tsg            tsl tu   w  wd  wg wgk   wgu  wk    wu   wx
0            default      cc9    Tuesday, November 5              90         Tue          C  Tue   0   0                   C  C   6 am    sunny    8       chart-sun             Mainly cloudy  Nov   1      pop3  30      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          9                0                NaN          -   0  cm   4   4    C   C  1572951600000  1572933600000  C  14   W  21  21  km/h  14  km/h  O-N
1            default      cc7    Tuesday, November 5              70         Tue          C  Tue   0   0                   C  C   7 am    sunny    3    chart-stormy   A mix of sun and clouds  Nov   2      pop3  30      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          7                0                NaN          -   0  cm   4   4    C   C  1572955200000  1572937200000  C  16   W  24  24  km/h  16  km/h    B
2            default      cc7    Tuesday, November 5              70         Tue          C  Tue   0   0                   C  C   8 am    sunny    3    chart-stormy   A mix of sun and clouds  Nov   3      pop3  30      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          7                0                NaN          -   0  cm   4   4    C   C  1572958800000  1572940800000  C  20   W  29  29  km/h  20  km/h    B
3            default      cc7    Tuesday, November 5              70         Tue          C  Tue   0   0                   C  C   9 am    sunny    3    chart-stormy   A mix of sun and clouds  Nov   4      pop3  30      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          7                0                NaN          -   0  cm   4   4    C   C  1572962400000  1572944400000  C  22   W  33  33  km/h  22  km/h    B
4            default      cc7    Tuesday, November 5              70         Tue          C  Tue   1   1                   C  C  10 am    sunny    3    chart-stormy   A mix of sun and clouds  Nov   5      pop2  20      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          7                0                NaN          -   0  cm   5   5    C   C  1572966000000  1572948000000  C  24   W  37  37  km/h  24  km/h    B
5            default      cc6    Tuesday, November 5              60         Tue          C  Tue   0   0                   C  C  11 am    sunny    3    chart-stormy   A mix of sun and clouds  Nov   6      pop2  20      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          6                0                NaN          -   0  cm   5   5    C   C  1572969600000  1572951600000  C  28   W  42  42  km/h  28  km/h    B
6            default      cc6    Tuesday, November 5              60         Tue          C  Tue   1   1                   C  C  12 pm    sunny    3    chart-stormy   A mix of sun and clouds  Nov   7      pop2  20      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          6                0                NaN          -   0  cm   6   6    C   C  1572973200000  1572955200000  C  29   W  44  44  km/h  29  km/h    B
7            default      cc6    Tuesday, November 5              60         Tue          C  Tue   3   3                   C  C   1 pm    sunny    3    chart-stormy   A mix of sun and clouds  Nov   8      pop2  20      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          6                0                NaN          -   0  cm   7   7    C   C  1572976800000  1572958800000  C  29   W  44  44  km/h  29  km/h    B
8            default      cc6    Tuesday, November 5              60         Tue          C  Tue   2   2                   C  C   2 pm    sunny    3    chart-stormy   A mix of sun and clouds  Nov   9      pop2  20      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          6                0                NaN          -   0  cm   6   6    C   C  1572980400000  1572962400000  C  27   W  41  41  km/h  27  km/h    B
9            default      cc7    Tuesday, November 5              70         Tue          C  Tue   2   2                   C  C   3 pm    sunny    3    chart-stormy   A mix of sun and clouds  Nov  10      pop3  30      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          7                0                NaN          -   0  cm   6   6    C   C  1572984000000  1572966000000  C  24   W  36  36  km/h  24  km/h    B
10           default      cc8    Tuesday, November 5              80         Tue          C  Tue   1   1                   C  C   4 pm    sunny    4  chart-overcast  Cloudy with sunny breaks  Nov  11      pop3  30      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          8                0                NaN          -   0  cm   5   5    C   C  1572987600000  1572969600000  C  23   W  35  35  km/h  23  km/h   B+
11           default      cc7    Tuesday, November 5              70         Tue          C  Tue   1   1                   C  C   5 pm    sunny   20  chart-overcast             Partly cloudy  Nov  12      pop3  30      -                0                NaN          -   0  mm      -  Tue Nov 5         False         False          7                0                NaN          -   0  cm   5   5    C   C  1572991200000  1572973200000  C  23   W  35  35  km/h  23  km/h   BN
12           ...