ELK堆栈中的IIS用户名
[英]IIS usernames in ELK stack
问题描述
我遇到一个问题，试图通过IIS日志和Elasticsearch解决。 发生的是，我在IIS日志中的用户名中包含一个反斜杠（），而不是一个正斜杠（/）。 当Elasticsearch返回名称时，它不再具有\\，我希望它在输入时会被转义。 因此，当在Elasticsearch或Kibana中查看结果时，用户名没有\\，并且那里的斜杠被视为正则表达式。 例如，用户名abcd \\ bob将作为abcdob返回。
我还认为，这就是我将_grokparsefailure标记添加到来自IIS的每个条目中的原因。
有什么建议么？
我的NXLOG文件获取数据：
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension json>
Module xm_json
</Extension>
#<Extension w3c>
#map iis log fields to Field Types
# Module xm_csv
# Fields $date, $time, $website, $serverip, $method, $url, $querystring, $port, $username, $clientip, $version, $useragent, $referer, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
# FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
#
# #Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
# #FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer
# Delimiter ' '
# QuoteChar '"'
# EscapeControl FALSE
# UndefValue -
#</Extension>
<Extension w3c>
#map iis log fields to Field Types
Module xm_csv
Fields $date, $time, $website, $hostname, $serverip, $verb, $request, $querystring, $dstport, $user, $clientip, $httpversion, $useragent, $cookie, $referrer, $fqdn, $status, $substatus, $sc_win32_status, $sc_bytes, $cs_bytes, $time_taken
FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer
Delimiter ' '
</Extension>
# Nxlog internal logs
<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
<Input iis-logs>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
ReadFromLast TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
to_json (); \
}
</Input>
<Output out>
Module om_tcp
Host logs.{domain removed}.com
Port 3515
</Output>
<Output iis-out>
Module om_tcp
Host logs.{domain removed}.com
Port 3516
</Output>
<Route 1>
Path internal, eventlog => out
</Route>
<Route 2>
Path iis-logs => iis-out
</Route>
我的Logstash.conf文件：
input {
tcp {
port => 5000
type => "syslog"
}
tcp {
type => "eventlog"
port => 3515
codec => json_lines
}
tcp {
type => "iislog"
port => 3516
codec => json_lines
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
# add_field => [ "received_from", "%{@source_host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}
if "_grokparsefailure" not in [tags] {
mutate {
replace => [ "@message", "%{syslog_message}" ]
}
}
mutate {
remove => [ "syslog_message", "syslog_timestamp" ]
}
kv {
source => "@message"
}
}
if [type] == "eventlog" {
# Incoming Windows Event logs from nxlog
# The EventReceivedTime field must contain only digits, or it is an invalid message
# if [EventReceivedTime] !~ /\d+/ { drop { } }
# grep {
# match => [ "EventReceivedTime", "\d+" ]
# }
mutate {
# Lowercase some values that are always in uppercase
lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
}
mutate {
# Set source to what the message says
rename => [ "Hostname", "@source_host" ]
}
date {
# Convert timestamp from integer in UTC
match => [ "EventReceivedTime", "UNIX" ]
}
mutate {
# Rename some fields into something more useful
rename => [ "Message", "@message" ]
rename => [ "Severity", "eventlog_severity" ]
rename => [ "SeverityValue", "eventlog_severity_code" ]
rename => [ "Channel", "eventlog_channel" ]
rename => [ "SourceName", "eventlog_program" ]
rename => [ "SourceModuleName", "nxlog_input" ]
rename => [ "Category", "eventlog_category" ]
rename => [ "EventID", "eventlog_id" ]
rename => [ "RecordNumber", "eventlog_record_number" ]
rename => [ "ProcessID", "eventlog_pid" ]
}
mutate {
# Remove redundant fields
remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
}
if [eventlog_id] == 4624 {
mutate {
add_tag => [ "ad-logon-success" ]
}
}
if [eventlog_id] == 4634 {
mutate {
add_tag => [ "ad-logoff-success" ]
}
}
if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 {
mutate {
add_tag => [ "ad-logon-failure" ]
}
}
if [eventlog_id] == 4723 {
mutate {
add_tag => [ "ad-password-change" ]
}
}
if [eventlog_id] == 4724 {
mutate {
add_tag => [ "ad-password-reset" ]
}
}
if "ad-logon-success" in [tags] {
metrics {
add_tag => [ "drop", "metric", "ad-logon-success" ]
meter => "ad-logon-success-metric"
}
}
if "ad-logon-failure" in [tags] {
metrics {
add_tag => [ "drop", "metric", "ad-logon-failure" ]
meter => "ad-logon-failure-metric"
}
}
}
if [type] == "iislog"
{
grok {
# match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{IP:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
match => ["message", "%{DATESTAMP:log_timestamp} %{WORD:sitename} %{HOSTNAME:computername} %{IP:hostip} %{URIPROTO:method} %{URIPATH:request} (?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{NOTSPACE:username}|-) %{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-) (?:%{HOSTNAME:host}|-) %{NUMBER:status} %{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received} %{NUMBER:bytes-sent} %{NUMBER:time-taken}"]
}
useragent {
source => "useragent"
}
#geoip {
# source => "clientip"
#}
}
metrics {
meter => "events"
add_tag => [ "drop", "metric", "events-metric" ]
}
}
output {
if "drop" not in [tags] {
elasticsearch {
host => "127.0.0.1"
cluster => "logs"
}
# stdout { codec => rubydebug }
}
}
IIS日志示例样本： 2015-05-06 15:41:18 W3SVC2 WEB1 10.11.10.137 GET /main/ - 80 ABCD\\smith 10.11.11.127 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.3;+WOW64;+Trident/7.0;+Touch;+.NET4.0E;+.NET4.0C;+Tablet+PC+2.0;+.NET+CLR+3.5.30729;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+InfoPath.3) cisession=a%3A4%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22959e25c7a1663350eeb85edb676de096%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%2210.11.11.127%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A120%3A%22Mozilla%2F4.0+%28compatible%3B+MSIE+7.0%3B+Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+Touch%3B+.NET4.0E%3B+.NET4.0C%3B+Tablet+PC+2.0%3B+.NET+CL%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1430926793%3B%7D08a40eacaf8b6eba6102c7746c35c46497a6502a http://my.domain.com/main/scheduling my.domain.com 200 0 0 11314 1009 1458
更新：我添加了第二个nxlog iis输入/输出，而不是将数据输出到我的logstash服务器，而是输出到平面文件。
<Output iis2-out>
Module om_file
File 'C:\logs\logtest.txt'
</Output>
我检查了此命令的输出，发现在击中Logstash服务器之前，用户名已从用户名中删除了反斜杠。
1 个解决方案
解决方案1
0 2015-05-11 22:20:44
我认为这是同样的问题。 您应该使用以下命令禁用默认的反斜杠转义符：
EscapeChar 0x00
配置ELK堆栈
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.