Python Request Waits not Responding

Question

import requests
res = requests.get("https://api.nasdaq.com/api/quote/list-type/nasdaq100")

When I enter url to browser it gives data, but when I use python it does not get data just waits. Thanks.

Answer 1

Seems like the user agent is the culprit after all:

>>> import requests
>>> requests.get("https://api.nasdaq.com/api/quote/list-type/nasdaq100", headers={"User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'})
<Response [200]>

Websites tend to block bots' and scrapers' user-agents. It is a common internet etiquette to respect their wishes.

Seems like Nasdaq's API has free and premium tiers. I guess they wish programmatic Python users to obtain an API key.

Keep in mind they also have their own Python API .

---

Looks like Nasdaq's server leaves the connection open and gets stuck when a different user agent is sent.

Try switching between the commented and uncommented ssl.send() :

import socket, ssl
context = ssl.create_default_context()
hostname = "api.nasdaq.com"
sock = socket.create_connection((hostname, 443))
ssl = context.wrap_socket(sock, server_hostname=hostname)
#ssl.send(b'GET /api/quote/list-type/nasdaq100 HTTP/1.1\r\nHost: api.nasdaq.com\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: identity\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n')
ssl.send(b'GET /api/quote/list-type/nasdaq100 HTTP/1.1\r\nHost: api.nasdaq.com\r\nUser-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X);AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75;Mobile/14E5239e Safari/602.1\r\nAccept-Encoding: identity\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n')
print(ssl.recv(1024))

Faking request's user agent in chrome works though and returns a 403 unauthorized.