Python - downloading application/csv data from webpage

Question

I'm using requests library to fetch a particular webpage which contains a link to download data in csv. The link is of the format

<a class="csv-download" download="data.csv" target"_blank"="" style="cursor:pointer" href="data:application/csv;charset=utf-8,%22Date%22%2C%22Volume%2FLength%22%2C%22Length%2FWidth%22%2C%22Weight%20gm%22%0A%2208-Jan-2018%22%2C%22%20%20%20%20%20%20%2023.19%22%2C%22%20%20%20%20%20%20%20%202.13%22%2C%22%20%20%20%20%20%20%20%201.32%22%0A" target="_blank">Download csv</a>

This link when clicked from the browser downloads the data in a file download.csv

I need to extract this as a csv and store to file. I'm using BeautifulSoup in the project for parsing HTML files.

How do I go about downloading the csv file from Python?

Here is what I have so far

import requests
from bs4 import BeautifulSoup as BS

r = requests.get(url)
soup = BS(r.text)
target_elt = soup.find('a', "csv-download")
# TODO - download the csv data

Answer 1

Since the contents of the file you need are stored in the href attribute of target_elt , starting after the comma, you can split the contents of that attribute on the first comma, then decode the portion after that first comma:

import urllib
import requests
from bs4 import BeautifulSoup as BS

r = requests.get(url)
soup = BS(r.text)
target_elt = soup.find('a', "csv-download")

header, encoded = target_elt.attrs["href"].split(",", 1)
data = urllib.unquote(encoded)
with open("data.csv", "w") as fp:
 fp.write(data)