IIS usernames in ELK stack
Question
I am having an issue that I am trying to resolve with IIS logs and Elasticsearch. What is happening is that my usernames in the IIS logs have a backslash () in them and not a forward slash (/). When Elasticsearch returns names it no longer has the \\ which I would have hoped would have been escaped when entered. So when viewing the results in Elasticsearch or Kibana the username has no \\ and the slash that is there is being treated as a regular expression. An example of this would be the username abcd\\bob would be returned as abcdob.
I also believe that this issue would be the reason I am getting a _grokparsefailure tag added to each entry coming from IIS.
Any suggestions?
My NXLOG file getting the data:
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension json>
Module xm_json
</Extension>
#<Extension w3c>
#map iis log fields to Field Types
# Module xm_csv
# Fields $date, $time, $website, $serverip, $method, $url, $querystring, $port, $username, $clientip, $version, $useragent, $referer, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
# FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
#
# #Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
# #FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
# Delimiter ' '
# QuoteChar '"'
# EscapeControl FALSE
# UndefValue -
#</Extension>
<Extension w3c>
#map iis log fields to Field Types
Module xm_csv
Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer
Delimiter ' '
</Extension>
# Nxlog internal logs
<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
<Input iis-logs>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
ReadFromLast TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
to_json (); \
}
</Input>
<Output out>
Module om_tcp
Host logs.{domain removed}.com
Port 3515
</Output>
<Output iis-out>
Module om_tcp
Host logs.{domain removed}.com
Port 3516
</Output>
<Route 1>
Path internal, eventlog => out
</Route>
<Route 2>
Path iis-logs => iis-out
</Route>
My Logstash.conf file:
input {
tcp {
port => 5000
type => "syslog"
}
tcp {
type => "eventlog"
port => 3515
codec => json_lines
}
tcp {
type => "iislog"
port => 3516
codec => json_lines
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
# add_field => [ "received_from", "%{@source_host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}
if "_grokparsefailure" not in [tags] {
mutate {
replace => [ "@message", "%{syslog_message}" ]
}
}
mutate {
remove => [ "syslog_message", "syslog_timestamp" ]
}
kv {
source => "@message"
}
}
if [type] == "eventlog" {
# Incoming Windows Event logs from nxlog
# The EventReceivedTime field must contain only digits, or it is an invalid message
# if [EventReceivedTime] !~ /\d+/ { drop { } }
# grep {
# match => [ "EventReceivedTime", "\d+" ]
# }
mutate {
# Lowercase some values that are always in uppercase
lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
}
mutate {
# Set source to what the message says
rename => [ "Hostname", "@source_host" ]
}
date {
# Convert timestamp from integer in UTC
match => [ "EventReceivedTime", "UNIX" ]
}
mutate {
# Rename some fields into something more useful
rename => [ "Message", "@message" ]
rename => [ "Severity", "eventlog_severity" ]
rename => [ "SeverityValue", "eventlog_severity_code" ]
rename => [ "Channel", "eventlog_channel" ]
rename => [ "SourceName", "eventlog_program" ]
rename => [ "SourceModuleName", "nxlog_input" ]
rename => [ "Category", "eventlog_category" ]
rename => [ "EventID", "eventlog_id" ]
rename => [ "RecordNumber", "eventlog_record_number" ]
rename => [ "ProcessID", "eventlog_pid" ]
}
mutate {
# Remove redundant fields
remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
}
if [eventlog_id] == 4624 {
mutate {
add_tag => [ "ad-logon-success" ]
}
}
if [eventlog_id] == 4634 {
mutate {
add_tag => [ "ad-logoff-success" ]
}
}
if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 {
mutate {
add_tag => [ "ad-logon-failure" ]
}
}
if [eventlog_id] == 4723 {
mutate {
add_tag => [ "ad-password-change" ]
}
}
if [eventlog_id] == 4724 {
mutate {
add_tag => [ "ad-password-reset" ]
}
}
if "ad-logon-success" in [tags] {
metrics {
add_tag => [ "drop", "metric", "ad-logon-success" ]
meter => "ad-logon-success-metric"
}
}
if "ad-logon-failure" in [tags] {
metrics {
add_tag => [ "drop", "metric", "ad-logon-failure" ]
meter => "ad-logon-failure-metric"
}
}
}
if [type] == "iislog"
{
grok {
# match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{IP:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
match => ["message", "%{DATESTAMP:log_timestamp} %{WORD:sitename} %{HOSTNAME:computername} %{IP:hostip} %{URIPROTO:method} %{URIPATH:request} (?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{NOTSPACE:username}|-) %{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-) (?:%{HOSTNAME:host}|-) %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
}
useragent {
source => "useragent"
}
#geoip {
# source => "clientip"
#}
}
metrics {
meter => "events"
add_tag => [ "drop", "metric", "events-metric" ]
}
}
output {
if "drop" not in [tags] {
elasticsearch {
host => "127.0.0.1"
cluster => "logs"
}
# stdout { codec => rubydebug }
}
}
Sample IIS log entry: 2015-05-06 15:41:18 W3SVC2 WEB1 10.11.10.137 GET /main/ - 80 ABCD\\smith 10.11.11.127 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.3;+WOW64;+Trident/7.0;+Touch;+.NET4.0E;+.NET4.0C;+Tablet+PC+2.0;+.NET+CLR+3.5.30729;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+InfoPath.3) cisession=a%3A4%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22959e25c7a1663350eeb85edb676de096%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2210.11.11.127%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A120%3A%22Mozilla%2F4.0+%28compatible%3B+MSIE+7.0%3B+Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+Touch%3B+.NET4.0E%3B+.NET4.0C%3B+Tablet+PC+2.0%3B+.NET+CL%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1430926793%3B%7D08a40eacaf8b6eba6102c7746c35c46497a6502a http://my.domain.com/main/scheduling my.domain.com 200 0 0 11314 1009 1458
UPDATE: I added a second nxlog iis input/output and instead of outputting the data to my logstash server I output to a flat file.
<Output iis2-out>
Module om_file
File 'C:\logs\logtest.txt'
</Output>
I checked the output of this and noticed that the username has the backslash removed from the username prior to hitting the logstash server.
1 answers
solution1
0 2015-05-11 22:20:44
I think this is the same issue. You should disable the default backslash escape character with:
EscapeChar 0x00
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.