How can I prevent read block from exiting while reading named pipe

Question

I have a BASH script that tails a file on an NFS mounted file-system and writes to a pipe called /tmp/t2.

#!/bin/bash
tail -f /mnt/cdr1/radius/calldatarecords | grep ^2 | awk -F, '{print $4","$13","$14","$15","$22","$23","$120","$140","$173}' | tr -d \" | tr - _  > /tmp/t2

output from mount

# mount
v7000:/ibm/cdr-fs on /mnt/cdr1 type nfs (ro,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.12.2.12,mountvers=3,mountport=32767,mountproto=udp,local_lock=none,addr=10.12.2.12,_netdev)

A second script reads from the blocking /tmp/t2 pipe

#!/bin/bash
echo "connected to pipe"
while read CID DUR CST DCD EGR ING PDD SGL MST
do
      echo Read Line at $(date +%s)
done < /tmp/t2
echo "what should never happen"

This has been running for years on Ub12.04 with only rare failures then it started happening frequently after a migration to Ub16.04 LTS. When coded initially I saw something of this nature when /mnt/cdr1/radius/calldatarecords was not being written to consistently at night. A while true around the failing read block solved that issue for the most part at that time, however that no longer works. What should never happen is frequent on 16.04. I've run both scripts under strace to capture what is happening. The second program exits with

write(1, "Read Line at 1522279814\n", 24Read Line at 1522279814
) = 24
ioctl(0, TCGETS, 0x7ffc48240330)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(0, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
read(0, "", 1)                          = 0
dup2(10, 0)                             = 0
fcntl(10, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0
read(255, "echo \"this should never happen\"\n", 158) = 32
write(1, "this should never happen\n", 25this should never happen
) = 25
read(255, "", 158)                      = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(0)                           = ?
+++ exited with 0 +++

and consequenly the first script strace exits reporting SIGPIPE. Or perhaps the the other way around, considering I don't recognize this data as part of the calldatarecords.

...
read(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 4424
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=12853, si_uid=0} ---
+++ killed by SIGPIPE +++

Can someone explain why the second program exits abnormally and how I can prevent it? Any help appreciated. Thanks for reading.

Answer 1

Try tailing your file with

tail --follow=name --retry

You might need to tune nfs mount as explained here .