Is it better to grep locally or remotely?

Question

I often log-in in a series of servers to grep for something in the logs.
I thought to automate this via a script to make it easier.
So my question is which approach is most efficient and sensible?
Doing

my @data = `ssh host grep pattern logfile`    

Or download the file locally (scp the file) and do the grep locally?

Answer 1

That depends on

• what percentage of log lines the grep filters
• the available CPU (remotely and locally)
• the available bandwidth for the scp
• the available filesystem size
• whether you may need the original logfile for other purposes subsequently

If you don't need the file later, and if doing the remote grep is not a problem, reducing the data before they are sent over the network (which is typically the bottleneck) is beneficial most of the time.

To further reduce the travelling data you can even do something like

my @data = `ssh host 'grep pattern logfile | gzip' | gunzip`

This should be mostly equivalent, for your purposes, to

my @data = `ssh -C host 'grep pattern logfile'`

although I have never tried this one.

BTW, you said you use egrep , which is fine, but I never get tired reminding people that grep interprets dots as wildcards, and therefore fgrep (or grep -F ) should be used for fixed strings.

Answer 2

The grep won't be "automatic" : it has to be performed by a computer somewhere.

The obvious answer is that it is best to do it remotely, because that way (potentially) fewer lines of data must be returned via the network. That is, unless the remote machine is so very slow that it is quicker to network the unwanted data than to filter it out.

Why don't you benchmark it to compare the two options? And don't start to optimise until you have things working well.

Answer 3

如果使用Java，则可以简单地使用grep4j库，否则，如果要图形化的内容，可以使用SSearcH Everywhere