Aggregating log files from remote linux servers in Java?

Question

I have never worked with reading files from linux machines and have no idea how to proceed. I have multiple linux servers which are accessible via ssh. Each machine has a log file which has some data, say some ip addresses. Now how do I write Java code to find the count of each ip address from these log files?

I read in another post that I can use JSch library to ssh to linux servers and read the file. But then how do I aggregate all the files? I'm assuming that copying these files in to local machine and then performing an aggregation would be very inefficient?

Answer 1

You can use exec.java example to start with. The command would be grep "ip address" /path/to/log_file | wc -l grep "ip address" /path/to/log_file | wc -l

However, if the log files are too big and grep is taking more resources, you will have to copy them locally and perform string search on them. In that case, you may want to start with scpfrom.java or sftp.java