如何批量缓冲和处理tail -f输出？

Question

I need to monitor a file and send what is written to it to a web service. I'm trying to achieve a clean and simple solution with bash scripting, eg:

#!/bin/bash

# listen for changes on file specified as first argument
tail -F "$1" | while read LINE
do
  curl http://service.com/endpoint --data "${LINE}"
done

This works perfectly, as in.. every line which is appended will be POST'ed to http://service.com/endpoint . However, I don't really like the fact that if many lines are appended in a short time, I will have as many HTTP requests and possibly overload the service.

Is there a smart way to kind of buffer the operations? I can think of something like:

buffer = EMPTY
while LINES are read:
  add LINE to buffer
  if buffer has more than X LINES
    send POST
  fi
done

But in the above solution if one line is posted per hour, I will only get updates every X hours, which is not acceptable. Another similar solution would be to "time" within the while loop: if X seconds have passed then send buffer, otherwise wait .. but the last line of a stream may be held indefinitely since the while loop is triggered only when a new line is added to the file.

The objective is to do this with minimal bash scripting and without using a second process . By second process I mean: process 1 gets the output from tail -f and stores it and process 2 periodically checks what is stored and sends a POST if more than x seconds are elapsed ?

I am curious if this is possible by some clever trick?

Thanks!

Answer 1

Literally putting your pseudocode to code:

# add stdbuf -oL if you care
tail -F "$1" | {
    # buffer = EMPTY
    buffer=
    # while LINES are read:
    while IFS= read -r line; do
      # add LINE to buffer
      buffer+="$line"$'\n'
      # if buffer has more than X LINES
      # TODO: cache the count of lines in a variable to save cpu
      if [ $(wc -l <<<"$buffer") -gt "$x_lines" ]; then
          # send POST
          # TODO: remove additional newline on the end of buffer, if needed
          curl http://service.com/endpoint --data "${buffer}"
          buffer=
      fi
    done
}

Removing the newline on the end of the buffer or for example buffering the number of lines in a separate counter to save cpu is left for others.

Notes:

• Uppercase variables by convention is reserved to global, exported variables.
• while read LINE will remove leading and trailing whitespaces from the line. Use while IFS= read -r line to read the whole line. More info at bashfaq how to read a file line by line
• With one line I believe you could just use xargs like tail -F "$1" | xargs -d$'\\n' -n1 curl http://service.com/endpoint --data tail -F "$1" | xargs -d$'\\n' -n1 curl http://service.com/endpoint --data

To buffer with the time, timeout the reading - either with bash extension, ex. read -t 0.1 or by timeouting the whole read timeout 1 cat .

To limit in both ways, the number of lines and with the timeout, I once written a badly named script called ratelimit.sh (badly named, because it does not limit rate...), that does exactly that. It reads lines, and if either count of lines or timeout is reached, it flushes it's buffer with additional output separator. I believe it's meant to be used like tail -F "$1" | ratelimit.sh --timeout=0.5 --lines=5 | while IFS= read -r -d $'\\x02' buffer; do curl ... --data "$buffer"; done tail -F "$1" | ratelimit.sh --timeout=0.5 --lines=5 | while IFS= read -r -d $'\\x02' buffer; do curl ... --data "$buffer"; done tail -F "$1" | ratelimit.sh --timeout=0.5 --lines=5 | while IFS= read -r -d $'\\x02' buffer; do curl ... --data "$buffer"; done . It roughly works like this:

# Written by Kamil Cukrowski (C) 2020
# Licensed jointly under MIT and Beerware license
# config
maxtimeoutns=$((2 * 1000 * 1000 * 1000))
maxlines=5 
input_separator=$'\n'
output_separator=$'\x02'

# the script
timeout_arg=()
while true; do
    chunk=""
    lines=0
    start=$(date +%s%N)
    stop=$((start + maxtimeoutns))

    while true; do

        if [ "$maxtimeoutns" != 0 ]; then
            now=$(date +%s%N)
            if (( now >= stop )); then
                break
            fi
            timeout=$(( stop - now ))
            timeout=$(awk -va=$timeout -vb=1000000000 '{print "%f", a/b}' <<<"")
            timeout_arg=(-t "$timeout")
        fi


        IFS= read -rd "$input_separator" "${timeout_arg[@]}" line && ret=$? || ret=$?

        if (( ret == 0 )); then

            # read succeded
            chunk+=$line$'\n'

            if (( maxlines != 0 )); then
                lines=$((lines + 1))
                if (( lines >= maxlines )); then
                    break
                fi
            fi

        elif (( ret > 128 )); then
            # read timeouted
            break;
        fi
    done

    if (( ${#chunk} != 0 )); then
        printf "%s%s" "$chunk" "$output_separator"
    fi

done

Answer 2

Thanks to KamilCuk 's answer, I managed to achieve what I wanted in a rather simple way, combining max number of lines and timeouts. The trick was to discover that the piping doesn't necessarily work by lines, like I thought it did..silly me!

Just for future reference this is my solution which is very specific and simplified to the bone:

#!/bin/bash
# sends updates to $1 via curl every 15 seconds or every 100 lines
tail -F "$1" | while true; do

    chunk=""
    stop=$((`date +%s` + 15))
    maxlines=100

    while true; do

        if (( `date +%s` >= stop )); then break; fi

        IFS= read -r -t 15 line && ret=$? || ret=$?         
        if (( ret == 0 )); then

                chunk+=$line$'\n'
                maxlines=$((maxlines - 1))
                if (( maxlines == 0 )); then break; fi

        elif (( ret > 128 )); then break; fi

    done

    if (( ${#chunk} != 0 )); then
        curl http://service.com --data "$chunk";
    fi

done