Community:Splunk for Change Management
From Splunk Wiki
Splunk for Change Management applies powerful indexing, search, alerting and reporting capabilities to the challenges of change management.
- ~30 reports
- ~38 saved searches
- 3 dashboard (status over view, trends, search overview)
A change management dashboard.
Installing the Splunk for Change Management Application
To install the Splunk for Change Management application, first unpack the tarball inside $SPLUNK_HOME/etc/apps. To configure this application, the issues you need to consider are data sources, host tags, event types, and alerts. The sections below address what needs to be done for each of these aspects. For more information, see the documentation on how to install Splunk applications.
Make sure you are editing the eventtypes to tag any type of change with the tag CM. The first eventtype in eventtypes.conf is meant to do this. You can change the eventtype to include other events that you want to have accounted for in summary statistics and reports.
File Change Monitor
Make sure you have an application installed (or you enable the input manually) that has the file system change monitor enabled. For example the UNIX application will do this. Monitor the files or directories on your system that you are interested in. Generally, on a UNIX machine you will want to monitor the /etc directory.
Make sure that if you index the actual files, the eventtype "config_file" is setup correctly for your environment. By default it assumes that all the files in /etc/ are configuration files. Change to your needs. If you are indexing network configuration files, for example with the Splunk for Networks Management application, make sure that your eventtype network_config is setup correctly. Both of these settings can be found in eventtypes.conf
To configure the ticketing integration, you have to make sure that all the events from your ticketing system are tagged with ticket. (The Splunk for Jira application is an example of an application that does this). That way the searches will pick those events up. Furthermore, make sure the following fields are populated:
- key the unique identifier for the ticket
- file the field in the ticket that identifies the file that has been changed or should be changed
- host_accepted the host that the change has been made on or the host where the change should be made
- created when the ticket was created, a timestamp of the format: %Y/%m/%d %T
- priority the priority of the ticket itself
If you are not running Splunk on your localhost, you have to change your field actions as follows:
- Make sure that there is a $SPLUNK_HOME/etc/apps/change_management/local directory.
- Copy $SPLUNK_HOME/etc/apps/change_management/default/field_actions.conf to the local directory.
- Edit the local/field_actions.conf file to replace localhost with the corresponding machine names.
If you want to see file changes with Splunk, you need to install Splunk on the systems where the changes occur. Splunk can be set up as lightweight forwarder which only forwards events and does not store or index them. See Forwarding and Receiving
To enable some of the data sources for change management, try installing the Splunk for UNIX application and enable the file system change monitors. Also enable the yum input on your linux (RedHat) systems. If you are interested in audit events, also enable the auditd input.
Use a classification schema of tags to flag the importance of your hosts. sev1 is used to flag critical machines. sev2 are a bit less critical. Down to sev4, which are not very important machines. You can use the stubs in eventtypes.conf to configure your hosts or interactively tag your hosts via the Web interface.
In addition to tagging the hosts with their severities, you should also tag them with their roles. Many searches are using a machine's role to detect variances in configurations. For example, tag your servers based on their role in the environment (i.e., Web, mail, ldap, etc.). The application can then detect changes in configurations between these machines. One of the limitations right now is that you should only tag each machine with one tag. The searches (e.g., CM-Summary-Variances by host class) makes use of that information.
In order to monitor policies, there are a few searches that you can use out of the box, for example the CM-policymonitor-httpd-listen-80 search. In order to monitor whether your Web server is listening on port 80, schedule this search and make sure the configuration file that contains this information (see search) is being indexed by splunk. If the specific line ("Listen 80") is found in the configuration file is found, it will be noted as a policy violation.
If you want to ad your own policy monitors for things like disallowed traffic, etc., you build your own search that follows the naming convention "CM-policymonitor-*". That way it will bubble up in the policy monitoring summaries.