How to read and format a stream of text received through a bash pipe?

Question

Currently, I'm using the following to format data from my npm script.

npm run startWin | while IFS= read -r line; do printf '%b\n' "$line"; done | less

It works, but my colleagues do not use Linux. So, I would like to implement while IFS= read -r line; do printf '%b\n' "$line"; done while IFS= read -r line; do printf '%b\n' "$line"; done while IFS= read -r line; do printf '%b\n' "$line"; done in Go, and use the binary in the pipe.

npm run startWin | magical-go-formater

What I tried

package main

import (
    "fmt"
    "io/ioutil"
    "os"
    "strings"
)

func main() {
  fi, _ := os.Stdin.Stat() // get the FileInfo struct

  if (fi.Mode() & os.ModeCharDevice) == 0 {

    bytes, _ := ioutil.ReadAll(os.Stdin)
    str := string(bytes)
    arr := strings.Fields(str)

    for _, v := range arr {
      fmt.Println(v)
    }
}

Currently the program silences any output from the text-stream.

Answer 1

You want to use bufio.Scanner for tail-type reads. IMHO the checks you're doing on os.Stdin are unnecessary, but YMMV.

See this answer for an example. ioutil.ReadAll() (now deprecated, just use io.ReadAll() ) reads up to an error/EOF, but it is not a looping input - that's why you want bufio.Scanner.Scan() .

Also - %b will convert any escape sequence in the text - eg any \n in a passed line will be rendered as a newline - do you need that? B/c go does not have an equivalent format specifier, AFAIK.

EDIT

So I think that, your ReadAll() -based approach would/could have worked...eventually. I am guessing that you were expecting behavior like you get with bufio.Scanner - the receiving process handles bytes as they are written (it's actually a polling operation - see the standard library source for Scan() to see the grimy details).

But ReadAll() buffers everything read and doesn't return until it finally gets either an error or an EOF. I hacked up an instrumented version of ReadAll() (this is an exact copy of the standard library source with just a little bit of additional instrumentation output), and you can see that it's reading as the bytes are written, but it just doesn't return and yield the contents until the writing process is finished, at which time it closes its end of the pipe (its open filehandle), which generates the EOF:

package main

import (
    "fmt"
    "io"
    "os"
    "time"
)

func main() {

    // os.Stdin.SetReadDeadline(time.Now().Add(2 * time.Second))

    b, err := readAll(os.Stdin)
    if err != nil {
        fmt.Println("ERROR: ", err.Error())
    }

    str := string(b)
    fmt.Println(str)
}

func readAll(r io.Reader) ([]byte, error) {
    b := make([]byte, 0, 512)
    i := 0
    for {
        if len(b) == cap(b) {
            // Add more capacity (let append pick how much).
            b = append(b, 0)[:len(b)]
        }
        n, err := r.Read(b[len(b):cap(b)])

        //fmt.Fprintf(os.Stderr, "READ %d - RECEIVED: \n%s\n", i, string(b[len(b):cap(b)]))
        fmt.Fprintf(os.Stderr, "%s READ %d - RECEIVED %d BYTES\n", time.Now(), i, n)
        i++

        b = b[:len(b)+n]
        if err != nil {
            if err == io.EOF {
                fmt.Fprintln(os.Stderr, "RECEIVED EOF")
                err = nil
            }
            return b, err
        }
    }
}

I just hacked up a cheap script to generate the input, simulating something long-running and writing only at periodic intervals, how I'd imagine npm is behaving in your case:

#!/bin/sh

for x in 1 2 3 4 5 6 7 8 9 10
do
  cat ./main.go
  sleep 10
done

As a side note, I find reading the actual standard library code really helpful...or at least interesting in cases like this.

Answer 2

@Sandy Cash was helpful in stating to use Bufio . I don't know why, if what @Jim said is true, but Bufio worked out and ReadAll() didn't.

Thanks for the help.

The code:

package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"
)

func main() {
    scanner := bufio.NewScanner(os.Stdin)
    for scanner.Scan() {
        s := scanner.Text()
        arr := strings.Split(s, `\n`)
        for _, v := range arr {
            fmt.Println(v)
        }
    }
}