Community:Monitoring a mixed sourcetype directory

From Splunk Wiki

Jump to: navigation, search

How to Monitor a mixed directory and sourcetyping by pattern

In splunk 4.1, 4.2, 4.3, and 5.x monitoring a mixed directory and sourcetyping by pattern is pretty trivial.

Overlapping directories works:



Also differently spelled stanzas with different patterns will work. For example here we index files ending in apache as sourcetype apache, and files ending in syslog as sourcetype syslog. if there are no other stanzas monitoring /logdirs, then other files in this directory will not be indexed.

sourcetype = apache

sourcetype = syslog

In splunk 3.x, 4.0, since overlapping path as parent and sub-directory, you cannot do this:

sourctype = syslog

sourcetype = apache

In splunk 3.x, and 4.0, since overlapping inputs aren't supported, You also can't do
(ref: )

sourcetype = apache

sourcetype = syslog

White/blacklists also don't solve this.

So instead, the solution is to capture all the files in the directory with one input stanza, and then break out the sourcetypes in props.conf:


# get files ending in .apache or .syslog
_whitelist = \.(apache|syslog)$


sourcetype = apache

sourcetype = syslog

# ..... breaks down into ... (anything) 
# and . (a literal '.' in the filename)

Another example: How to assign sourcetypes for different paths in props.conf

# How to assign sourcetypes for different paths in props.conf

# Reference:

- Setting up multiple sourcetypes
(ref: )

- How wild card (* or ...) are translated in inputs.conf
(ref: )

# Condition:  

-  We'd like to monitor following files



- And, specify different sourcetypes based on source paths 

    <sourcetype = ssmadmin_alert>

    <sourcetype = ssmadmin_debug>

    <sourcetype = ssmserver_alert>

    <sourcetype = ssmserver_debug>

# Solution:

1. @UF/LWF, edit inputs.conf and props.conf
- inputs.conf
# Note: 
#  ***Do not specify a sourcetype in this stanza in order to use props.conf to specify sourcetypes***
whitelist =  (SSMAdmin|SSMServer)[^/]*/(alerts|debug).\log$
index = mytest

- props.conf
# Note:
#     *** This is on the UF, not Indexer ***
sourcetype = ssmadmin_alert

sourcetype = ssmadmin_debug

sourcetype = ssmserver_alert

sourcetype = ssmserver_debug

2. @Indexer, search result
(Note: source paths are shortened)
Preview of: index=mytest | stats count by source, sourcetype

         source             sourcetype      count
------------------------- ---------------   -----
.../SSMServer1/alerts.log ssmserver_alert    13
.../SSMServer1/debug.log  ssmserver_debug    16
.../SSMAdmin1/alerts.log  ssmadmin_alert     14
.../SSMAdmin1/debug.log   ssmadmin_debug     31
Personal tools
Hot Wiki Topics

About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk