How to find most frequent user agent in nginx access.log

Question

In order to counter a botnet attack, I am trying to analyze a nginx access.log file to find which user agents are the most frequent, so that I can find the culprits and deny them. How can I do that?

Answer 1

Try something like this on your access log, replace with the path to your access log, also keep in mind that some log files would get zipped and new one would be created

sudo awk -F" " '{print $1}' /var/log/nginx/access.log | sort | uniq -dc

EDIT :

Sorry I just noticed you wanted user agent instead of IP

sudo awk -F"\"" '{print $6}' /var/log/nginx/access.log | sort | uniq -dc

To sort ascending append | sort -nr | sort -nr and to limit to 10 append | head -10 | head -10

so the final total line would be

sudo awk -F"\"" '{print $6}' /var/log/nginx/access.log | sort | uniq -dc | sort -nr | head -10

Answer 2

To get user agent

sudo awk -F'"' '/GET/ {print $6}' /var/log/nginx-access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn


awk(1) - selecting full User-Agent string of GET requests
cut(1) - using first word from it
sort(1) - sorting
uniq(1) - count
sort(1) - sorting by count, reversed