Splunk is Monitoring Its own Logs

Question

We are trying to monitor log files on a server using the Splunk universal forwarder. The logs directory (say /logs/app3/Oct2015) is being monitored by Splunk forwarder.

The Splunk documentation tells that the use of blacklist can be done to stop monitoring of any unwanted directory. Accordingly, my input.conf is here with: ( Splunk Documentation )

/opt/splunkforwarder/etc/system/local/inputs.conf :

[monitor:///logs/app3/Oct2015/]
    sourcetype = access_common
    ignoreOlderThan = 7d
    recurse = true
    _TCP_ROUTING = toLogAggregator
    blacklist = /opt/splunkforwarder/...

[blacklist:/opt/splunkforwarder/...]

This is my Output configuration -

/opt/splunkforwarder/etc/system/local/outputs.conf : ( Splunk Documentation on Output )

[tcpout]
defaultGroup = toBeIgnored

# This target group will redirect events to Log aggregator listening on TCP socket.
[tcpout:toLogAggregator]
server = 10.20.176.207:9997
sendCookedData = false

[tcpout:toBeIgnored]

When I run the Splunk Universal Forwarder (v6.3), I am successfully able to monitor my logs directory, but Splunk is also sending me its own logs. I added a blacklist stanza to stop Splunk from monitoring and sending me its own logs, but no luck. My aggregator server is continuously receiving Splunk logs mixed with logs I am monitoring.

I added a TCP output group to drain events by default and explicitly marked my monitored directory to route logs using a different TCP group, but that also didn't work. Still getting bombarded with Splunk logs.

Does anyone knows, how to tel Splunk Forwarder to not to monitor itself but only monitor the directory I have requested for ?

regards,

-Vipul;

Answer 1

There was a plugin present in /opt/splunkforwarder/etc/apps/ by default, called SplunkUniversalForwarder . Inside the plugin directory, there is a default directory that contains an inputs.conf file. This file contains the monitor statements for $SPLUNK_HOME/var/log folder.

I disabled them and Bingo, Splunk stopped flooding the destination with Splunk logs, while sending what I asked Splunk to forward.

All the details are here: https://answers.splunk.com/answers/320050/how-to-tell-a-splunk-universal-forwarder-to-not-to.html