Web request works in Browser but gives HTTP 403 in Java

Question

I am running a Java Program to download data from a Stock API. The URL is

https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918

If I run this URL from the browser I get a JSON as a response. However if I run it from a Java program I get a 403 forbidden. I looked up the Developer tools in Chrome and setup all the HTTP Request Headers in the Java program. I added User-Agent/Cookie etc etc. However I still get the Forbidden 403 error.

Below are the HTTP Headers that I have set:

Below is the stacktrace that I get in my Java program.

java.io.IOException: Server returned HTTP response code: 403 for URL: https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1997)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
    at java.base/java.net.URLConnection.getContent(URLConnection.java:753)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getContent(HttpsURLConnectionImpl.java:404)
    at ai.jeet.test.DataDownload.main(DataDownload.java:36)

When I run this API call via Postman I get a forbideen 403 but also get a HTML response which says enable Cookies and Javascript.

I fail to understand how a Server can differentiate between a Web Request call. Obviously I am doing something wrong but I can't figure out what:-(

Answer 1

The main difference from what I could see is the User Agent header value, the server might be checking for few specific header values, I have one value working for me as shown below -

Request with 403 -

    curl -v 'https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918' -H 'User-Agent: Mozilla' -o /dev/null

Response -

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700::6812:9a:443...
* Connected to tvc4.investing.com (2606:4700::6812:9a) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [323 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2326 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=investing.com
*  start date: Aug  1 00:00:00 2022 GMT
*  expire date: Aug  1 23:59:59 2023 GMT
*  subjectAltName: host "tvc4.investing.com" matched cert's "*.investing.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x137012600)
> GET /49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918 HTTP/2
> Host: tvc4.investing.com
> accept: */*
> user-agent: Mozilla
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403
< date: Wed, 23 Nov 2022 14:22:29 GMT
< content-type: text/html; charset=UTF-8
< cache-control: max-age=15
< expires: Wed, 23 Nov 2022 14:22:44 GMT
< x-frame-options: SAMEORIGIN
< set-cookie: __cf_bm=uwoeNXvcnmEDY7ACRnFQEDqGZN4Yfx2_cyXbho.D6.M-1669213349-0-AUqobigV2idaMayGrBR+OdyTBo8pbjfS77vjhSh6bA4wiaBuz79/5kbwvXwD2loYoHJt1BsTguMEYh7WRm2ikPo=; path=/; expires=Wed, 23-Nov-22 14:52:29 GMT; domain=.investing.com; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 76ea8eab8cbbb06a-ATL
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
{ [972 bytes data]
100  4537    0  4537    0     0  23760      0 --:--:-- --:--:-- --:--:-- 25346
* Connection #0 to host tvc4.investing.com left intact

Request with 200 Response -

    curl -v 'https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918' -H 'User-Agent: Mozilla/5.0' -o /dev/null

Response -

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700::6812:19a:443...
* Connected to tvc4.investing.com (2606:4700::6812:19a) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [323 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2326 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=investing.com
*  start date: Aug  1 00:00:00 2022 GMT
*  expire date: Aug  1 23:59:59 2023 GMT
*  subjectAltName: host "tvc4.investing.com" matched cert's "*.investing.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x13800b600)
> GET /49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918 HTTP/2
> Host: tvc4.investing.com
> accept: */*
> user-agent: Mozilla/5.0
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Wed, 23 Nov 2022 14:22:13 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: https://tvc-invdn-com.investing.com
< x-requested-with: XMLHttpRequest
< access-control-allow-methods: POST, GET, OPTIONS, PUT, DELETE
< access-control-allow-headers: Content-Type, Depth, User-Agent, X-File-Size, X-Requested-With, If-Modified-Since, X-File-Name, Cache-Control, accept, sessionid, x-csrftoken, content-type
< x-benchmark-1a: 0ms, mem alloc - 768.00Kb start
< x-benchmark-1b: 3ms, mem alloc - 1024.00Kb getPairIdBySymbol
< x-benchmark-1c: 3ms, mem alloc - 1024.00Kb getIntervalByResolution
< x-benchmark-1d: 15ms, mem alloc - 1024.00Kb getPairDataAndAttributes in TradingviewConnector::findCandles
< x-benchmark-1e: 23ms, mem alloc - 3.50Mb getCandles in TradingviewConnector::findCandles
< x-benchmark-1f: 24ms, mem alloc - 3.75Mb end of TradingviewConnector::findCandles
< x-benchmark-1g: 25ms, mem alloc - 3.50Mb findCandles
< vary: Accept-Encoding,User-Agent
< content-security-policy: upgrade-insecure-requests; block-all-mixed-content
< cf-cache-status: DYNAMIC
< set-cookie: __cf_bm=z7PZf80TAcglrPBqnAC0p4ApGjbAfRvCP6uZDBinzzs-1669213333-0-AYMziboPajXlvpP44SA/dGfK246VFdm8eaSnco06Ug7FiLfKcTheS1UjW4S8yD0EEJFSW7kqengoBx6dH8W7YLg=; path=/; expires=Wed, 23-Nov-22 14:52:13 GMT; domain=.investing.com; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 76ea8e461cc9ad68-ATL
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
{ [314 bytes data]
100 19882    0 19882    0     0  61780      0 --:--:-- --:--:-- --:--:-- 62719
* Connection #0 to host tvc4.investing.com left intact

Answer 2

Server cant really differentiate between Postman call or Browser call. In this case it seems like the application is using cookie based authentication. If you configure everything properly in Postman/Java code it should return you the information without 403 status code.

Cookie-Based Authentication:

Cookies based Authentication normally works in these 4 steps-

1. The user provides a username and password in the login form and the client/browser sends a login request.

2. After the request is made, the server validates the user on the backend by querying the database. If the request is valid, it will create a session by using the user information fetched from the database and store them. For each session a unique ID called the session ID is created. By default, the session ID will be
   given to the client through the Browser.

3. The Browser will submit this session ID on each subsequent request. 4. The session ID is verified against the database. Based on this session ID, the server will identify the session belonging to which client and then give the request access. Once a user logs out of the app, the session is destroyed both client-side and server-side.

May be you are missing specific cookie. Check this article about setting the cookies properly in postman.: https://www.toolsqa.com/postman/cookies-in-postman/