使用Logstash Grok过滤器过滤系统日志消息

Question

I have the following logs sent to logstash via syslog input (the logs come from docker containers sent to logstash via logspout)

So, in the following log sample, the first line is a php-fpm error and the second line is a nginx access log

docker-nginx-php-composer|2015/03/17 16:31:28 [error] 9#0: *41 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Call to undefined function teetstst() in /var/www/index.php on line 3" while reading response header from upstream, client: 10.10.37.110, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "turing:49213"
docker-nginx-php-composer|10.10.37.110 - - [17/Mar/2015:16:31:28 +0100] "GET /favicon.ico HTTP/1.1" 200 134 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36"

```

OK so now I want to filter all this mess so I can "tag" PHP-FPM logs for example or filter only the warning and errors of PHP-FPM ...

I suppose I have to use the grok filter from logstash but I don't know exactly how ?

let say I want to filter only PHP errors, put a "php" tag on it and also filter 404 and 500 errors from nginx, how can I achieve this easily with logstash/grok ?

Any clues ? examples ?

Thanks :)

Answer 1

I would define two grok patterns:

1. php-fpm that starts with a date
2. nginx that starts with an ip

You can add_tag in each grok{} as desired.