在 Splunk 上过滤访问日志

Question

SplunkForwarder is used in order to provide the Apache's access log to Splunk (or was told by DevOps so). AFAIK it's not possible to filter out logs based on given regEx -- the ISSUE that I'm trying to solve. Was thinking to add a trigger on Apache that will intercept all requests and send a message to Splunk if the URL pattern of the request is in the whitelist (have found Splunk HTTP Event Collector - never used before - smells like a part of the solution). Tried to find a proper example of how to use mod_actions module. Unfortunately, didn't find anything that works for me. The Apache's documentation is useless. Not sure whether mod_actions is the only option I have. Pls, could you tell me how can I execute py/sh/pl or any other script for each request? Likely, DevOps won't allow me to add any exotic (non-standard) module to Apache. Thanks a lot.

Answer 1

It's totally possible to filter logs based on regex. You'll need an app on indexer to do filtering based on the sourcetype defined in inputs.conf on the universal forwarder.

props.conf/transforms.conf combo provided below will keep only the log lines containing strings Error or Warning while all other messages will be sent to nullqueue (deleted).

$SPLUNK_HOME$/etc/apps/your_app/local/props.conf $SPLUNK_HOME$/etc/apps/your_app/local/transforms.conf

props.conf

[your_sourcetype]
TRANSFORMS-set = setnull, setparsing

transforms.conf

# This sends all events to be ignored
[setnull] 
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

# this says ignore all events, except the ones containing ERROR
[setparsing]
REGEX = Error|Warning
DEST_KEY = queue
FORMAT = indexQueue

[your_sourcetype1]
TRANSFORMS-set = setnull, setparsing

[your_souretype2]
TRANSFORMS-set = setnull, setparsing