[Bash]从txt文件中读取URL

Question

I have a text file and I would like to read URLs from this text file one by one and check if they expire within 30 days or more.

Exame URLS:

example01.example.com:8080
example02.example.com:8002
example03.example.com:8003
https://example04.example.com:8111
...

I have a Bash script which checks an SSL cert's expiry date:

 #!/bin/bash
TARGET="example01.example.com:8080";
DAYS=30;
echo "checking if $TARGET expires in less than $DAYS days";
expirationdate=$(date -d "$(: | openssl s_client -connect $TARGET -servername $TARGET 2>/dev/null \
                              | openssl x509 -text \
                              | grep 'Not After' \
                              |awk '{print $4,$5,$7}')" '+%s'); 
time=$(($(date +%s) + (86400*$DAYS)));
if [ $time -gt $expirationdate ]; then
    echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d @$expirationdate '+%Y-%m-%d')" ;
    echo $TARGET on $(date -d @$expirationdate '+%Y-%m-%d') > expireEndpoint.txt;
else
    echo "OK - Certificate expires on $(date -d @$expirationdate '+%Y-%m-%d')";
fi;

Some of the URLs are duplicates too.

Answer 1

slightly hacky but I came up with this quick and dirty solution... here is my take

EDIT:fixed a small bug

> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
    do
        if [ -z "$line" ]; then
            continue   # ignore if line is empty
        fi

        protocol=`echo $line | awk -F: '{print $1}'` 
        temp=(${line//:/ })
        PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
        temp=`echo $line | awk -F/ '{print $3}'`
        TARGET=`echo $temp | awk -F: '{print $1}'`
        DAYS=30;

        # 1. Check if it is uses secure HTTP
        if [ $protocol == "https" ]; then
            echo "$TARGET is secure: uses HTTPS"
        else
            echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
            echo "$TARGET uses HTTP: insecure" 
        fi

        # 2. check if server is reachable
        if curl --output /dev/null --silent --head --fail "$TARGET"; then
            echo "$TARGET is reachable "
        else
            echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
            echo "============" >> unreachableOrInsecure.txt
            echo "$TARGET is unreachable " 
            echo "============"
            continue   # do not continue anymore, check the next target
        fi

        # 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
        echo "checking if $TARGET:$PORT expires in less than $DAYS days";
        expirationdate=$(date -d "$(: | timeout 3 openssl s_client -connect $TARGET:$PORT -servername $TARGET 2>/dev/null \
                              | openssl x509 -text \
                              | grep 'Not After' \
                              |awk '{print $4,$5,$7}')" '+%s'); 
        time=$(($(date +%s) + (86400*$DAYS)));

        if [ $time -gt $expirationdate ]; then
            echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d @$expirationdate '+%Y-%m-%d')" ;
            echo $TARGET on $(date -d @$expirationdate '+%Y-%m-%d') >> certLessThan30Days.txt;
            echo "============" >> certLessThan30Days.txt
        else
            echo "OK - Certificate expires on $(date -d @$expirationdate '+%Y-%m-%d')";
            echo $TARGET on $(date -d @$expirationdate '+%Y-%m-%d') >> certMoreThan30Days.txt;
            echo "============" >> certMoreThan30Days.txt
        fi;

        echo "============"
    done < "$input"

EDIT: adding the url.txt and the output Here is my url.txt

    https://msn.com:443/services/f?a
    https://google.com:443
    http://example.com:80/c/ca/s
    https://www.google.ie/ 
    https://www.facebook.com/ 
    https://www.kooba.ie/

output of the script

msn.com is secure: uses HTTPS
msn.com is reachable 
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on 2022-04-06
============
google.com is secure: uses HTTPS
google.com is reachable 
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on 2020-09-22
============
example.com uses HTTP: insecure
example.com is reachable 
checking if example.com:80 expires in less than 30 days
unable to load certificate
140024794944832:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for example.com expires in less than 30 days, on 2020-07-23
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable 
checking if www.google.ie: expires in less than 30 days
unable to load certificate
139738534053184:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.google.ie expires in less than 30 days, on 2020-07-23
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable 
checking if www.facebook.com: expires in less than 30 days
unable to load certificate
139623076582720:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.facebook.com expires in less than 30 days, on 2020-07-23
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable 
checking if www.kooba.ie: expires in less than 30 days
unable to load certificate
139831347127616:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.kooba.ie expires in less than 30 days, on 2020-07-23
============

Answer 2

modified the above script to fix the openssl error

> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
    do
        if [ -z "$line" ]; then
            continue   # ignore if line is empty
        fi

        protocol=`echo $line | awk -F: '{print $1}'` 
        temp=(${line//:/ })
        PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
        temp=`echo $line | awk -F/ '{print $3}'`
        TARGET=`echo $temp | awk -F: '{print $1}'`
        DAYS=30;

        # 1. Check if it is uses secure HTTP
        if [ $protocol == "https" ]; then
            echo "$TARGET is secure: uses HTTPS"
        else
            echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
            echo "$TARGET uses HTTP: insecure" 
        fi

        # 2. check if server is reachable
        if curl --output /dev/null --silent --head --fail "$TARGET"; then
            echo "$TARGET is reachable "
        else
            echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
            echo "============" >> unreachableOrInsecure.txt
            echo "$TARGET is unreachable " 
            echo "============"
            continue   # do not continue anymore, check the next target
        fi

        # 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
        echo "checking if $TARGET:$PORT expires in less than $DAYS days";
        expirationdate=$(timeout 3  openssl s_client -servername $TARGET -connect $TARGET:$PORT </dev/null 2>/dev/null | \
                         openssl x509 -noout -dates 2>/dev/null | \
                         awk -F= '/^notAfter/ { print $2; exit }')
        expire_epoch=$(date +%s -d "$expirationdate")
        epoch_warning=$(($DAYS*86400)) #look for 30 days
        today_epoch="$(date +%s)"
        timeleft=`expr $expire_epoch - $today_epoch`

        if [[ $timeleft -le $epoch_warning ]]; then 
            echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d @$expire_epoch)" ;
            echo "Cert expires for $TARGET on $(date -d @$expire_epoch '+%Y-%m-%d')" >> certLessThan30Days.txt;
            echo "============" >> certLessThan30Days.txt
        else
            echo "OK - Certificate expires on $(date -d @$expire_epoch)";
            echo "Cert expires for $TARGET on $(date -d @$expire_epoch '+%Y-%m-%d')" >> certMoreThan30Days.txt;
            echo "============" >> certMoreThan30Days.txt
        fi;

        echo "============"
    done < "$input"

Output now looks like this

msn.com is secure: uses HTTPS
msn.com is reachable 
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on Thursday 07 April 2022 02:18:44 AM IST
============
google.com is secure: uses HTTPS
google.com is reachable 
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on Wednesday 23 September 2020 02:13:12 AM IST
============
example.com uses HTTP: insecure
example.com is reachable 
checking if example.com:80 expires in less than 30 days
KO - Certificate for example.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable 
checking if www.google.ie: expires in less than 30 days
KO - Certificate for www.google.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable 
checking if www.facebook.com: expires in less than 30 days
KO - Certificate for www.facebook.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable 
checking if www.kooba.ie: expires in less than 30 days
KO - Certificate for www.kooba.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============