python sys.stdin.read() from tail -f

Question

How come sys.stdin.read() doesn't read the piped input from tail -f?

#!/usr/bin/env python
import sys
from geoip import geolite2
def iplookup(srcip):
        for ip in srcip.split("\n"):
                try:
                        print(geolite2.lookup(ip))
                except:
                        pass
source = sys.stdin.read()
iplookup(source)

tail -f /var/log/bleh.log | grep -oE '((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])' | python mygeoip.py

Answer 1

None of the other answers (even fileinput ) fully addresses the issue of buffering, and so will not work for small outputs of tail -f .

From the python man page :

Note that there is internal buffering in xreadlines(), readlines() and file-object iterators ("for line in sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a "while 1:" loop.

In other words what you want is:

while True:
    line = sys.stdin.readline()
    iplookup(line)

Answer 2

You can use fileinput :

import sys
from geoip import geolite2
import fileinput

def iplookup(srcip):
        for ip in srcip.split("\n"):
                try:
                        print(geolite2.lookup(ip))
                except:
                        pass

for line in fileinput.input():
    iplookup(line)

On the plus side, your script automagically accepts filename as parameters as well.

Answer 3

You can use sys.stdin as an iterator, rather than trying to read from it first.

def iplookup(srcip):
    for ip in srcip:
        ip = ip.strip()
        try:
            print(geolite2.lookup(ip))
        except:
            pass

iplookup(sys.stdin)

Answer 4

read() reads until EOF is reached. EOF char is added when close() is performed or you can add it explicitly.

Your file does not have any EOF. Modify your program to read blocks of fixed size or iterate over leadline() instead.