如何在Logstash中正确使用Nginx自定义日志

Question

I've found the good article about using of nginx custom log format for logstash. In one on topic comment there is:

Be careful: Between two (referrer, user agent) and four (request method, remote_user) can be set by the end-user and thus can cause the JSON to be invalid (ie set the user-agent to "}).

As long as nginx doesn't have explicit JSON support, I would recommend against manually >building JSON and stick to combined log which is well-supported by logstack.

How should I use custom log format to prevent this issue?

Answer 1

Create grok with content

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}

And filter

filter {
    grok {
        patterns_dir => ["Grok_DIR"]
        match => { "message" => "%{NGINXACCESS}" }
    }
}

Answer 2

It is not bulletproof in any version of nginx to hand craft the JSON - \\xXX is not valid JSON encoding, it should be \\u00XX. Any proper JSON parser will reject the JSON string if nginx escapes with \\xXX.

If you do this with LogStash you will mostly be OK but expect json parse failures from time to time.

Answer 3

I think it's fine in recent versions of nginx:

Changes with nginx 1.1.6 17 Oct 2011

*) Change: now the 0x7F-0x1F characters are escaped as \xXX in an
   access_log.

Changes with nginx 0.7.0 19 May 2008

*) Change: now the 0x00-0x1F, '"' and '\' characters are escaped as \xXX
   in an access_log.
   Thanks to Maxim Dounin.