ANSI escape sequences breaking when called by a script

Question

I've been using a program called fzf for selecting things in scripts for some time but on several occasions I have found myself trying to run these scripts on systems which does not have this program installed.

To remedy this I wrote a mini clone (But obviously far slower on large data sets) I can embed in my scripts to avoid the dependency entirely.

It uses ANSI escape sequences to "summon a second terminal" (I don't remember what it's called) and avoid filling the regular one with junk. The script works fine if I call it manually, but it completely breaks if I call it in a subshell or function. It also spits the escape sequences into the next program if I pipe the output or run it in a subshell.

At this point I have spent many hours trying to debug it and I'm not sure how to continue. Is there some other way I am supposed to use these sequences if called in a script.

Here is the code:

#!/bin/bash
# FZY: Command Line Fuzzy Finder
# Created: 29/10/2020
# Author: Nan0Scho1ar (Christopher Mackinga)
# License: MIT License


input="$(< /dev/stdin)"; height="$(tput lines)"; inum=$(echo "$input" | wc -l)
echo -e "\e[?1049h" 1>&2; row=1; col=1; str=""; cur=1; fnum=$inum; regex=""; regex2=""
while true; do
    range="$row,$((row+height-3))p;$((row+height-3))q"
    filtered=$(echo "$input" | grep ".*$regex" | sed -n $range 2>/dev/null | sed -e 's/.*/  &/');
    frange="$(echo "$filtered" | wc -l)"
    curpos=$((frange-cur+1))
    echo "$filtered" | cut -c$col- | grep -E --color=always "$regex2" | tac | sed $curpos's/^  \(.*\)/> \1/'
    echo "  $fnum/$inum" && read -r -p "> $str" -n 1 -s < /dev/tty && read -r -sn3 -t 0.001 k1 < /dev/tty; REPLY+=$k1;
    case "$REPLY" in
        '') echo -e "\e[?1049l" 1>&2; echo "$filtered" | sed -n "${cur}s/  //p;${cur}q;" && exit;;
        $'\e[C'|$'\e0C') col=$((col+1));;
        $'\e[D'|$'\e0D') [[ $col -gt 1 ]] && col=$((col-1));;
        $'\e[B'|$'\e0B') [[ $cur -ge 1 ]] && cur=$((cur-1));;
        $'\e[A'|$'\e0A') [[ $cur -le $fnum ]] && cur=$((cur+1));;
        $'\e[1~'|$'\e0H'|$'\e[H') row=1;;
        $'\e[4~'|$'\e0F'|$'\e[F') row=$fnum;;
        *)
            char=$(echo "$REPLY" | hexdump -c | awk '{ print $2 }');
            if [[ $char = "033" ]]; then echo -e "\e[?1049l" 1>&2 && break;
            elif [[ $char = "177" ]] && [[ ${#str} -gt 0 ]]; then str="${str::-1}";
            else str="$str$REPLY" && row=1; fi
            regex=$(echo "$str" | sed "s/\(.\)/\1.*/g");
            regex2=$(echo "$str" | sed "s/\(.\)/\1|/g");
            fnum=$(echo "$input" | grep -c ".*$regex")
        ;;
    esac
    [[ $((frange-cur+1)) -lt 1 ]] && row=$((row+1)) && cur=$((cur-1));
    [[ $cur -lt 1 ]] && row=$((row-1)) && cur=$((cur+1));
    [[ $cur -gt $fnum ]] && cur=$fnum;
    [[ $((row-fnum+frange)) -gt 1 ]] && row=$((row-1))
    [[ $row -lt 1 ]] && row=1;
    yes '' | sed "${height}q";
done

An example of a usage which works:

echo 'test 1
test 2
test 3' | fzy

An example of a usage which does not work:

output=$(echo 'test 1
test 2
test 3' | fzy)

EDIT: As raised in comments, not capturing the output in from the subshell was not representative of my actual use case. Updated example failing case.

EDIT2: redirecting control sequences to stderr seems to have improved things. updated code to reflect this change.

Answer 1

The solution to my problem was to send all the control sequences and menu/UI related lines to stderr as Charles Duffey suggested in the comments. All credit to him for pointing this out.

Here is the final code with the changes applied. (There are a still a couple minor bugs but they are completely unrelated to this question.)

Note a few things such as echo -ne "\e[?1049h" were changed to echo -ne "\e[?1049h" 1>&2

#!/bin/bash
# FZY: Command Line Fuzzy Finder
# Created: 29/10/2020
# Author: Nan0Scho1ar (Christopher Mackinga)
# License: MIT License

input="$(< /dev/stdin)"; height="$(tput lines)"; inum=$(echo "$input" | wc -l)
echo -ne "\e[?1049h\r" 1>&2; row=1; col=1; str=""; cur=1; fnum=$inum; regex=""; regex2=""
while true; do
    range="$row,$((row+height-3))p;$((row+height-3))q"
    filtered=$(echo "$input" | grep ".*$regex" | sed -n $range 2>/dev/null | sed -e 's/^.*/  &/');
    frange="$(echo "$filtered" | wc -l)"
    curpos=$((frange-cur+1))
    echo "$filtered" | cut -c$col- | grep -E --color=always "$regex2" | tac | sed $curpos's/^  \(.*\)/> \1/' 1>&2
    echo "  $fnum/$inum" 1>&2 && read -r -p "> $str" -n 1 -s < /dev/tty && read -r -sn3 -t 0.001 k1 < /dev/tty; REPLY+=$k1;
    case "$REPLY" in
        '') echo -ne "\e[?1049l" 1>&2; echo "$filtered" | sed -n "${cur}s/  //p;${cur}q;" && exit;;
        $'\e[C'|$'\e0C') col=$((col+1));;
        $'\e[D'|$'\e0D') [[ $col -gt 1 ]] && col=$((col-1));;
        $'\e[B'|$'\e0B') [[ $cur -ge 1 ]] && cur=$((cur-1));;
        $'\e[A'|$'\e0A') [[ $cur -le $fnum ]] && cur=$((cur+1));;
        $'\e[1~'|$'\e0H'|$'\e[H') row=1;;
        $'\e[4~'|$'\e0F'|$'\e[F') row=$fnum;;
        *)
            char=$(echo "$REPLY" | hexdump -c | awk '{ print $2 }');
            if [[ $char = "033" ]]; then echo -e "\e[?1049l" 1>&2 && exit 1;
            elif [[ $char = "177" ]] && [[ ${#str} -gt 0 ]]; then str="${str::-1}";
            else str="$str$REPLY" && row=1; fi
            regex=$(echo "$str" | sed "s/\(.\)/\1.*/g");
            regex2=$(echo "$str" | sed "s/\(.\)/\1|/g");
            fnum=$(echo "$input" | grep -c ".*$regex")
        ;;
    esac
    [[ $((frange-cur+1)) -lt 1 ]] && row=$((row+1)) && cur=$((cur-1));
    [[ $cur -lt 1 ]] && row=$((row-1)) && cur=$((cur+1));
    [[ $cur -gt $fnum ]] && cur=$fnum;
    [[ $((row-fnum+frange)) -gt 1 ]] && row=$((row-1))
    [[ $row -lt 1 ]] && row=1;
    yes '' | sed "${height}q" 1>&2;
done