curl 返回 HTTP 状态码 404 但页面可用

Question

I want to check the HTTP status code for a web UI (https://something/login.html) with curl on Linux. So I wrote a little shell script doing the following:

#!/bin/sh
echo $(curl -sSLk -w '%{http_code}' -o /dev/null $1 2> /dev/null)

The problem is: this script returns 404 even if the queried web UI is available through the browser. For other URLs the script returns valid values (status code 200).

I further wrote a Python script for testing reasons:

import requests
try:
   response = requests.get(
       "https://something/login.html",
       verify=False
   )   
   print(response.status_code)
except requests.ConnectionError:
   print("failed to connect")

this Python script returns status code 200 for the same URL for which curl returns 404.

Thanks for your help in advance.

Answer 1

They detected what you were trying to do. Some sites return a Forbidden or Not Authoriozed.
Any web app can return any HTTP Status Code.

Just using a UA is not sufficient. Using an obscure UA that cannot be profiled may (or may not) help. It has worked for me a few times.

CloudFlare is using the UA for profiling. One of the things they do is watch the SSL handshaking. Browsers do not all use the same algorithm and CloudFlare finds the idiosyncrasies between various Browsers. You have to have all the cookies. Many will do a 304 redirect back to them selves after setting some cookie and check for the cookie. If you do not have JavaScript you can be blocked.


Without the URL I cannot tell you much. There is a learning curve, and a fairly steep one.

If I can enter a site using a Browser without JavaScript, I can always get in with curl.
Some are still very difficult. I like the ones that put all the data I want in a JavaScript JSON object.