finding -k value of decrypted file (openssl)

Question

Replaced old code with correct one.

The main problem turned out to be that the file type was of ISO-8859 and not "ASCII" or "ansi" probably due to the file being generated on a windows computer. It was also encrypted with an older version of openssl. The newer version uses SHA-256 while the old one uses MD5 if I'm not mistaken.

#!/bin/bash

for i in {0000..9999}
do
    if openssl enc -aes-256-cbc -d -md md5 -in encrypted.dta -out decrypted.txt -k $i 2>/dev/null && file decrypted.txt |grep ': ISO-8859'
    then
    echo $i:; cat decrypted.txt;break
    fi
done

Answer 1

The relatively few (slightly more than 1/256) cases where openssl enc -d of a padded cipher does not give an error for a wrong password, it will produce essentially garbage output, but that will be the correct length or nearly so. IF the plaintext was ASCII (no chars with eighth bit set) there is very low probability a non-correct output will do the same, and that's easy to check:

for i in {0000..9999}; do 
  if openssl enc -aes-256-cbc -d -in 65891080.enc -out 65891080.tmp 2>/dev/null -k $i && 
    file 65891080.tmp | grep -q ': ASCII text'
  then echo $i:; cat 65891080.tmp; fi
done
6543:
This is the secret text