Get/fetch a file with a bash script using /dev/tcp over https without using curl, wget, etc

Question

I try to read/fetch this file:

https://blockchain-office.com/file.txt with a bash script over dev/tcp without using curl,wget, etc..

I found this example:

exec 3<>/dev/tcp/www.google.com/80
echo -e "GET / HTTP/1.1\r\nhost: http://www.google.com\r\nConnection: close\r\n\r\n" >&3
cat <&3

I change this to my needs like:

exec 3<>/dev/tcp/www.blockchain-office.com/80
echo -e "GET / HTTP/1.1\r\nhost: http://www.blockchain-office.com\r\nConnection: close\r\n\r\n" >&3
cat <&3

When i try to run i receive:

400 Bad Request
Your browser sent a request that this server could not understand

I think this is because strict ssl/only https connections is on.

So i change it to :

exec 3<>/dev/tcp/www.blockchain-office.com/443
echo -e "GET / HTTP/1.1\r\nhost: https://www.blockchain-office.com\r\nConnection: close\r\n\r\n" >&3
cat <&3

When i try to run i receive:

400 Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

So i even can't get a normal connection without get the file!

All this post's does not fit, looks like ssl/tls is the problem only http/80 works, if i don't use curl, wget, lynx, openssl, etc...:

how to download a file using just bash and nothing else (no curl, wget, perl, etc.)

Using /dev/tcp instead of wget

How to get a response from any URL?

Read file over HTTP in Shell

I need a solution to get/read/fetch a normal txt file from a domain over https only with /dev/tcp no other tools like curl, and output in my terminal or save in a variable without wget, etc.., is it possible and how, or is it there an other solution over the terminal with the standard terminal utilities?

Answer 1

You can use openssl s_client to perform the equivalent operation but delegate the SSL part:

#!/bin/sh

host='blockchain-office.com'
port=443
path='/file.txt'

crlf="$(printf '\r\n_')"
crlf="${crlf%?}"

{
  printf '%s\r\n' \
    "GET ${path} HTTP/1.1" \
    "host: ${host}" \
    'Connection: close' \
    ''
} |
  openssl s_client -quiet -connect "${host}:${port}" 2 >/dev/null | {

  # Skip headers by reading up until encountering a blank line
  while IFS="${crlf}" read -r line && [ -n "$line" ]; do :; done

  # Output the raw body content
  cat
}

Instead of cat to output the raw body, you may want to check some headers like Content-Type , Content-Transfer-Encoding and even maybe navigate and handle recursive MIME chunks, then decode the raw content to something.

Answer 2

After all the comments and research, the answer is no, we can't get/fetch files using only the standard tools with the shell like /dev/tcp because we can't handle ssl/tls without handle the complete handshake.

It is only possbile with the http/80 .

i dont think bash's /dev/tcp supports ssl/tls

If you use /dev/tcp for a http/https connection you have to manage the complete handshake including ssl/tls, http headers, chunks and more. Or you use curl/wget that manage it for you.

then shell is the wrong tool because it is not capable of performing any of the SSL handshake without using external resources/commands. Now relieve and use what you want and can from what I show you here as the cleanest and most portable POSIX-shell grammar implementation of a minimal HTTP session through SSL. And then maybe it is time to consider alternative options (not using HTTPS, using languages with built-in or standard library SSL support).

We will use curl , wget and openssl on seperate docker containers now.

I think there are still some requirements in the future to see if we keep only one of them or all of them.

We will use the script from @Léa Gris in a docker container too.