Getting 500 Response on Python Requests

Question

import requests

url = "https://uk.eu-supply.com/ctm/supplier/publictenders?B=BLUELIGHT"

payload={'Branding': 'BLUELIGHT',
'SearchFilter.BrandingCode': 'BLUELIGHT',
'CpvContainer.CpvIds': '',
'CpvContainer.CpvIds': '',
'SearchFilter.PagingInfo.PageNumber': '2',
'SearchFilter.PagingInfo.PageSize': '25'}
files=[

]
headers = {
  'Cookie': 'EUSSESSION=e3cc7bc4-ea51-4c4b-99f7-2ebde589c8e0'
}

response = requests.request("POST", url, headers=headers, data=payload, files=files)

print(response.status_code)

eu-supply.com
https://uk.eu-supply.com

I am trying to hit request to this site, the same payload and url request works with postman. Notice (two same fields on payload "CpvContainer.CpvIds ", this is must. In order to get 200 response , we need to pass this field twice. But as dict can only set one key unique. I am getting 500.

Answer 1

To read a table from the page, simple pd.read_html is enought:

import pandas as pd

url = "https://uk.eu-supply.com/ctm/supplier/publictenders?B=BLUELIGHT"

df = pd.read_html(url)[0]
print(df)

Prints:

    Quote/tender Id           Reference                                                                                                           Name Date of publication Response deadline (UK time)                                                Process                                            Buyers       Countries
0             45024       7F- 2020-0278                                           Kennelling of Seized & Dangerous Dogs (Kent Police and Essex Police)          25/06/2021            26/07/2021 12:00                         Above Threshold OPEN Procedure                              7 Forces Procurement  United Kingdom
1             45074            CONT0056                                                                                Leaflets for Crime Commissioner          25/06/2021            05/07/2021 12:00               Tender/Quotation (below Threshold value)                             Leicestershire Police  United Kingdom
2             45072        7F-2021-P038                                                            Bury St Edmunds Police Station - Car Park Extension          25/06/2021            26/07/2021 12:00    Formal Tender/Quotation (Advertised) (over £50,000)                              7 Forces Procurement  United Kingdom
3             44234    7F - 2020 - 0387                                                                      Fire Doors Inspection and Maintenance PMA          24/06/2021            26/07/2021 10:00                         Above Threshold OPEN Procedure                              7 Forces Procurement  United Kingdom
4             45068        7F-2021-P040                                                           Armoury Improvements- Bury St Edmunds Police Station          24/06/2021            26/07/2021 12:00    Formal Tender/Quotation (Advertised) (over £50,000)                              7 Forces Procurement  United Kingdom
5             45042    BLC-Aviation-003                                  Provision of Police Aviation Services including a Fleet Replacement Programme          23/06/2021            06/07/2021 12:00            Prior Information Notice (PIN) (Standalone)                              BlueLight Commercial  United Kingdom
6             42896              DP0570                                                          Reactive Mechanical Services and Small Works Projects          23/06/2021            03/08/2021 12:00                         Above Threshold OPEN Procedure  The Police and Crime Commissioner for Derbyshire  United Kingdom
7             45043    BLC-Aviation-003  Market Engagement Event for the Provision of Police Aviation Services including a Fleet Replacement Programme          23/06/2021            06/07/2021 12:00           Request for Information /Market Consultation                              BlueLight Commercial  United Kingdom
8             44914          20-21 EM15                                                                          Asbestos Management Services Contract          22/06/2021            20/07/2021 12:00  Formal Tender/Quotation (Advertised) ( up to £50,000)     Tyne and Wear Fire and Rescue Service (TWFRS)  United Kingdom
9             44793           1363-2016                                                                  Recovery of loose, stray and abandoned horses          21/06/2021            21/07/2021 14:00            Prior Information Notice (PIN) (Standalone)                 West Yorkshire Combined Authority  United Kingdom
10            44734        7F 2020 0339                                                                               Bodyshop Replacement and Repairs          21/06/2021            23/07/2021 12:00                         Above Threshold OPEN Procedure                              7 Forces Procurement  United Kingdom
11            44984                 NaN                                    Integrated Communications Control System (ICCS) for Northamptonshire Police          18/06/2021            30/06/2021 17:00            Prior Information Notice (PIN) (Standalone)                      MINT Commercial Services LLP  United Kingdom
12            44983             WMP0126                                                         Garage Equipment - Maintenance, Calibration and Repair          18/06/2021            19/07/2021 12:00               Tender/Quotation (below Threshold value)                              West Midlands Police  United Kingdom
13            44962  DHR Chair & Author                                                                        Domestic Homicide Review Chair & Author          17/06/2021            05/07/2021 12:00            Prior Information Notice (PIN) (Standalone)                              7 Forces Procurement  United Kingdom
14            44950                1197                                        Access Control/Automated Security Gate Maintenance and Related Services          17/06/2021            23/07/2021 14:00    Formal Tender/Quotation (Advertised) (over £50,000)                    Royal Berkshire Fire Authority  United Kingdom
15            44959             627_RFI                                                                      Evidence Based Practice Partnership Board          17/06/2021            16/07/2021 17:00           Request for Information /Market Consultation     Chief Constable for Devon and Cornwall Police  United Kingdom
16            44944                W344                                                                            Repair & Maintenance of UPS Systems          16/06/2021            13/07/2021 12:00               Tender/Quotation (below Threshold value)                                Northumbria Police  United Kingdom
17            44942        7F-2021-C058                                                                       Specialist Accredited Fire Investigation          16/06/2021            30/06/2021 12:00           Request for Information /Market Consultation                              7 Forces Procurement  United Kingdom
18            44916       WYP/CS/019/JB                                                                                          Fire Risk Assessments          16/06/2021            30/06/2021 14:00  Formal Tender/Quotation (Advertised) ( up to £50,000)                 West Yorkshire Combined Authority  United Kingdom
19            44594        7F-2020-0273                                                                      Norfolk Integrated Domestic Abuse Service          15/06/2021            20/07/2021 12:00                         Above Threshold OPEN Procedure                              7 Forces Procurement  United Kingdom
20            44911                SPCC                                    Sussex PCC RfQ - An evaluation for the domestic abuse perpetrator programme          15/06/2021            30/06/2021 12:00  Formal Tender/Quotation (Advertised) ( up to £50,000)  Office of the Sussex Police & Crime Commissioner  United Kingdom
21            44543           2420-2021       Force Medical Advisor (FMA) / Selected Medical Practitioner (SMP) and other Occupational Health Services          15/06/2021            14/07/2021 13:00                         Above Threshold OPEN Procedure                 West Yorkshire Combined Authority  United Kingdom
22            44845             C003196                                                     THERAPEUTIC INTERVENTIONS AND TRAINING PROVISION FOR WMFRA          15/06/2021            06/07/2021 16:00    Formal Tender/Quotation (Advertised) (over £30,000)           West Midlands Fire and Rescue Authority  United Kingdom
23            44544            ESFA0199                                                          CMI Leadership and Management Online Training Courses          14/06/2021            09/07/2021 12:00    Formal Tender/Quotation (Advertised) (over £25,000)                        East Sussex Fire Authority  United Kingdom
24            44714                 590                                                        Regional Grounds Maintenance and Winter Salting Service          11/06/2021            15/07/2021 12:00                         Above Threshold OPEN Procedure     Chief Constable for Devon and Cornwall Police  United Kingdom

Answer 2

Make the repeated key's values as a list:

payload={
...
'CpvContainer.CpvIds': ['',''],
...
}

Answer 3

In the above API use the payload as raw text. I cross-checked the below code is successfully fetching the data

import requests

url = "https://uk.eu-supply.com/ctm/Supplier/publictenders/PublicTenders"

payload="Branding=BLUELIGHT&SearchFilter.BrandingCode=BLUELIGHT&CpvContainer.CpvIds=&CpvContainer.CpvIds=&SearchFilter.PagingInfo.PageNumber=2&SearchFilter.PagingInfo.PageSize=25"
headers = {
  'Connection': 'keep-alive',
  'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"',
  'Accept': '*/*',
  'X-Requested-With': 'XMLHttpRequest',
  'sec-ch-ua-mobile': '?0',
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36',
  'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
  'Origin': 'https://uk.eu-supply.com',
  'Sec-Fetch-Site': 'same-origin',
  'Sec-Fetch-Mode': 'cors',
  'Sec-Fetch-Dest': 'empty',
  'Referer': 'https://uk.eu-supply.com/ctm/supplier/publictenders?B=BLUELIGHT',
  'Accept-Language': 'en-US,en;q=0.9',
  'Cookie': 'EUSSESSION=b18ce90d-32b9-429c-8d74-7151a997e8cd'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)