python script for downloading file from website

Question

I am trying to download csv file from a website. Only when I click on the url print(link) url gets opened in new tab and file gets downloaded.

Need help

1. How do I code so that to enable download of that file into my script itself? (
2. How can I give specific location for download?

import requests
import urllib.request
from bs4 import BeautifulSoup



headers = {
    'Connection': 'keep-alive',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
    'Accept': 'text/html, */*; q=0.01',
    'X-Requested-With': 'XMLHttpRequest',
    'sec-ch-ua-mobile': '?0',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'Origin': 'https://www.samco.in',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Dest': 'empty',
    'Referer': 'https://www.samco.in/bhavcopy-nse-bse-mcx',
    'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
}

data = [
  ('start_date', '2021-05-24'),
  ('end_date', '2021-05-30'),
  ('show_or_down', '1'),
  #('bhavcopy_data[]', 'NSE'),
  ('bhavcopy_data[]', 'NSEFO'),
  #('bhavcopy_data[]', 'BSE'),
  #('bhavcopy_data[]', 'MCX'),
]

response = requests.post('https://www.samco.in/bse_nse_mcx/getBhavcopy', headers=headers,data=data)
print(response)
soup = BeautifulSoup(response.text, 'html.parser')

one_a_tag = soup.findAll('a')[3]

# one_a_tag = soup.findAll('a')
link = one_a_tag['href']
print(link)

Answer 1

You should send a GET request to the new link that you have obtained:

link = one_a_tag['href']
r = requests.get(link)

You can then get the contents of the file in bytes and Unicode using r.content and r.text:

file_contents = r.content
file_text = r.text

In your case, since you want to store comma-separated values in python, you can read the values into a dataframe using pandas:

import pandas as pd
from io import StringIO
df = pd.read_csv(StringIO(file_contents.decode('utf-8')), error_bad_lines = False)

You can also write the Unicode contents to a file:

open('something.csv', 'w').write(file_text)