ELK堆栈中的IIS用户名

Question

I am having an issue that I am trying to resolve with IIS logs and Elasticsearch. What is happening is that my usernames in the IIS logs have a backslash () in them and not a forward slash (/). When Elasticsearch returns names it no longer has the \\ which I would have hoped would have been escaped when entered. So when viewing the results in Elasticsearch or Kibana the username has no \\ and the slash that is there is being treated as a regular expression. An example of this would be the username abcd\\bob would be returned as abcdob.

I also believe that this issue would be the reason I am getting a _grokparsefailure tag added to each entry coming from IIS.

Any suggestions?

My NXLOG file getting the data:

## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.

    #define ROOT C:\Program Files\nxlog
    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension json>
        Module      xm_json
    </Extension>

    #<Extension w3c>
      #map iis log fields to Field Types
    #    Module      xm_csv 
    #    Fields      $date, $time, $website, $serverip, $method, $url, $querystring, $port, $username, $clientip, $version, $useragent, $referer, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
    #    FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
    #    
    #    #Fields      $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
    #    #FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
    #    Delimiter   ' '
    #   QuoteChar '"'
    #   EscapeControl FALSE
    #   UndefValue -
    #</Extension>

    <Extension w3c>
      #map iis log fields to Field Types
        Module      xm_csv
        Fields      $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
        FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer
        Delimiter   ' '
    </Extension>


    # Nxlog internal logs
    <Input internal>
       Module im_internal
       Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
    </Input>


    # Windows Event Log
    <Input eventlog>
    # Uncomment im_msvistalog for Windows Vista/2008 and later 
       Module im_msvistalog

    # Uncomment im_mseventlog for Windows XP/2000/2003
    #   Module im_mseventlog

       Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
    </Input>

    <Input iis-logs>
        Module      im_file
        File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
        ReadFromLast TRUE
        Exec        if $raw_event =~ /^#/ drop();                    \
                    else                                             \
                    {                                                \
                        w3c->parse_csv();                            \
                        $EventTime = parsedate($date + " " + $time); \
                        to_json ();                                  \
                    }
    </Input>


    <Output out>
       Module om_tcp
       Host logs.{domain removed}.com
       Port 3515
    </Output>

    <Output iis-out>
        Module      om_tcp
        Host        logs.{domain removed}.com
        Port        3516
    </Output>


    <Route 1>
       Path internal, eventlog => out
    </Route>

    <Route 2>
        Path        iis-logs => iis-out
    </Route>

My Logstash.conf file:

input {
        tcp {
            port => 5000
            type => "syslog"
        }
        tcp {
            type   => "eventlog"
            port   => 3515
            codec => json_lines
        }
        tcp {
            type => "iislog"
            port => 3516
            codec => json_lines
        }
    }

    filter {
        if [type] == "syslog" {
            grok {
                match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" }
                add_field => [ "received_at", "%{@timestamp}" ]
    #           add_field => [ "received_from", "%{@source_host}" ]
            }
            syslog_pri { }
            date {
                match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
            }
            if "_grokparsefailure" not in [tags] {
                mutate {
                    replace => [ "@message", "%{syslog_message}" ]
                }
            }
            mutate {
                remove => [ "syslog_message", "syslog_timestamp" ]
            }
            kv {
                source => "@message"
            }
        }
        if [type] == "eventlog" {
            # Incoming Windows Event logs from nxlog
            # The EventReceivedTime field must contain only digits, or it is an invalid message
    #       if [EventReceivedTime] !~ /\d+/ { drop {  } }
    #       grep {
    #           match => [ "EventReceivedTime", "\d+" ]
    #       }
            mutate {
                # Lowercase some values that are always in uppercase
                lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
            }
            mutate {
                # Set source to what the message says
                rename => [ "Hostname", "@source_host" ]
            }
            date {
                # Convert timestamp from integer in UTC
                match => [ "EventReceivedTime", "UNIX" ]
            }
            mutate {
                # Rename some fields into something more useful
                rename => [ "Message", "@message" ]
                rename => [ "Severity", "eventlog_severity" ]
                rename => [ "SeverityValue", "eventlog_severity_code" ]
                rename => [ "Channel", "eventlog_channel" ]
                rename => [ "SourceName", "eventlog_program" ]
                rename => [ "SourceModuleName", "nxlog_input" ]
                rename => [ "Category", "eventlog_category" ]
                rename => [ "EventID", "eventlog_id" ]
                rename => [ "RecordNumber", "eventlog_record_number" ]
                rename => [ "ProcessID", "eventlog_pid" ]
            }
            mutate {
                # Remove redundant fields
                remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
            }
            if [eventlog_id] == 4624 {
                mutate {
                    add_tag => [ "ad-logon-success" ]
                }
            }
            if [eventlog_id] == 4634 {
                mutate {
                    add_tag => [ "ad-logoff-success" ]
                }
            }
            if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 {
                mutate {
                    add_tag => [ "ad-logon-failure" ]
                }
            }
            if [eventlog_id] == 4723 {
                mutate {
                    add_tag => [ "ad-password-change" ]
                }
            }
            if [eventlog_id] == 4724 {
                mutate {
                    add_tag => [ "ad-password-reset" ]
                }
            }
            if "ad-logon-success" in [tags] {
                metrics {
                    add_tag => [ "drop", "metric", "ad-logon-success" ]
                    meter => "ad-logon-success-metric"
                }
            }
            if "ad-logon-failure" in [tags] {
                metrics {
                    add_tag => [ "drop", "metric", "ad-logon-failure" ]
                    meter => "ad-logon-failure-metric"
                }
            }
        }
        if [type] == "iislog"
        {
            grok {
    #    match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{IP:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
                match => ["message", "%{DATESTAMP:log_timestamp} %{WORD:sitename} %{HOSTNAME:computername} %{IP:hostip} %{URIPROTO:method} %{URIPATH:request} (?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{NOTSPACE:username}|-) %{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-) (?:%{HOSTNAME:host}|-) %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
            }
            useragent {
                source => "useragent"
            }
            #geoip {
            #   source => "clientip"
            #}
        }
        metrics {
            meter => "events"
            add_tag => [ "drop", "metric", "events-metric" ]
        }
    }

    output {
        if "drop" not in [tags] {
            elasticsearch {
            host => "127.0.0.1"
            cluster => "logs"
        }
    #       stdout { codec => rubydebug }
        }
    }

Sample IIS log entry: 2015-05-06 15:41:18 W3SVC2 WEB1 10.11.10.137 GET /main/ - 80 ABCD\\smith 10.11.11.127 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.3;+WOW64;+Trident/7.0;+Touch;+.NET4.0E;+.NET4.0C;+Tablet+PC+2.0;+.NET+CLR+3.5.30729;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+InfoPath.3) cisession=a%3A4%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22959e25c7a1663350eeb85edb676de096%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2210.11.11.127%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A120%3A%22Mozilla%2F4.0+%28compatible%3B+MSIE+7.0%3B+Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+Touch%3B+.NET4.0E%3B+.NET4.0C%3B+Tablet+PC+2.0%3B+.NET+CL%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1430926793%3B%7D08a40eacaf8b6eba6102c7746c35c46497a6502a http://my.domain.com/main/scheduling my.domain.com 200 0 0 11314 1009 1458

UPDATE: I added a second nxlog iis input/output and instead of outputting the data to my logstash server I output to a flat file.

<Output iis2-out>
        Module      om_file
        File 'C:\logs\logtest.txt'
    </Output>

I checked the output of this and noticed that the username has the backslash removed from the username prior to hitting the logstash server.

Answer 1

I think this is the same issue. You should disable the default backslash escape character with:

EscapeChar 0x00