Installing Splunk in the Enterprise Step by Step
From Splunk Wiki
This guide is for help with the overall tasks needed to install Splunk in a Distributed Deployment suitable for the Enterprise, e.g. an Enterprise Security Use Case
The following guide has been assembled to provide a checklist for and considerations for the Installation and Configuration of Enterprise Security. This guide is NOT an authoritative or complete guide, see docs.splunk.com for latest and authoritative reference, here at latest Enterprise Security manual 
Splunk provides detailed documentation on each subject and we strongly encourage all Splunk Administrators to read the documentation relevant to the topic at hand at docs.splunk.com as the final reference and latest information. When you get stuck, Splunk has a large free support infrastructure that can help:
- Splunk Answers 
- Splunk Docs 
- Splunk Community http://blogs.splunk.com/tag/community/
- Splunk Community Wiki 
- The Splunk Internet Relay Chat (IRC) channel (EFNet #splunk). (IRC client required)
(Chrome try the Kiwi Extension or Mibbit) If you still don't have an answer to your question, you can get in touch with Splunk's support by opening a case. When you open a case please include a diag file to help with troubleshooting from the affected system.
Before Starting Here are the KEY references you will need: Review these links for the latest Splunk App for Enterprise Security information:
- ES manual http://docs.splunk.com/Documentation/ES/latest/Install/Overview
- Splunk Education for ES Training - CIRT / CSOC etc. http://www.splunk.com/view/SP-CAAAH9S
Before Installing Enterprise Security - Splunk Core:
- Develop a Splunk data collection topology with Splunk Professional Services and Support, you can get help with all of these items and more. Contact Splunk Professional Services.
Have a Splunk Core Deployment in place: Make sure hardware or virtual machines are sized for the deployment and install operating systems.
- See the "Hardware capacity planning for your Splunk deployment" in the Splunk documentation
- Install OS of choice, decide on the mount points for your warm, cold and frozen data.
- Verify Network connectivity, ports 
- Create a local Splunk account and a splunk domain user for system activities.
Ensure Splunk Admin has installed or downloaded the necessary software.
* Binaries of Splunk Enterprise, Splunk Universal Forwarder, Splunk Enterprise Security * Designate and Setup Contacts for your Support Entitlement (accessible on customer¹s account page) * Splunk license keys (accessible on customer¹s account page) * Ability to Collect and Submit Support Health Check Diagnostic files * http://docs.splunk.com/Documentation/Splunk/6.0/Troubleshooting/ContactSplunkSupport * http://answers.splunk.com/answers/2115/remotely-pull-splunk-diag-via-rest * http://docs.splunk.com/Documentation/Splunk/6.0.2/Troubleshooting/AnonymizedatasamplestosendtoSupport#Linux_tip:_Anonymize_all_log_files_from_a_diag_at_once
ES is typically setup in a distributed Splunk deployment which consists of different systems which are dedicated to running Splunk, configured in the following roles;
- Splunk Indexer(s)
- Splunk Search Head for License Master, Deployment Server, and other Apps
- Splunk Dedicated Forwarder on Windows OS for if Microsoft Data Sources
- Splunk Dedicated Forwarder on Linux for Syslog, and appliance data sources
- Splunk Search Head for only Enterprise Security App.
Review these links for the latest Splunk Core Deployment information: Splunk deployment http://docs.splunk.com/Documentation/ES/latest/Install/DeploymentPlanning 
Splunk Universal Forwarder Download http://www.splunk.com/en_us/download/universal-forwarder.html 
Splunk Enterprise Download http://www.splunk.com/en_us/download/splunk-enterprise.html 
Planning to collect Enterprise Security Data Sources
- Review Current log collection capabilities and goals
- Understand the data sources that are required and recommended to make the most meaningful correlations for security content for your organization.
- Consider Data Sources for perimeters like firewalls, core routers, etc.
- Consider Data Sources for internal appliances like IDS, IPS, DNS, ActiveDirectory, LDAP
- Consider Collecting windows logs, by deploying out the Splunk Universal Forwarder to Windows Servers
- Consider other operating system logs, e.g. Linux, Solaris, and deploy a Splunk Universal Forwarder
- Consider the destination of syslog traffic sources to a log management system.
- Consider Data sources for Assets and Identities *** critical to using Splunk workflow
- Assets and Identities http://docs.splunk.com/Documentation/ES/latest/User/Identitymanagement
Assets and Identities Lists and Feeds
Plan and develop the Assets and Identities feeds with attention to identify the known/Expected devices and hosts These are key files to make sure are filled out to the best of the ability of the system owner.
This includes understanding your enterprise's assets and identities on the "blue" team defenders side. With an authoritative source of valid assets (servers, workstations, phones, etc.) and valid identities (admins, guests, users) then an understanding of the enterprise's overall security posture can be established.
Assets (blue team systems) : Think about DMZ, crown jewels, ftp systems, vendor gateways, financial servers, HR servers Identities (blue team access) : Think about privileged user groups, administrators, services accounts, vendor accounts, ftp accounts
- You need at minimum the IP addresses of the systems you want to gather data from and enter these into this csv file.
- Plan to have a script run from a cmdb or a network nMAP scan to collect this data regularly about the environment
Plan to survey server owners on the impact of data compromise of their systems. Establish criticality of assets by bunit.
Note: This default set of column headers must be in any asset file you use. ip,mac,nt_host,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should_timesync,should_update,requires_av
identities.csv You need at minimum a list of known default, privileged, service and administrator accounts. All members of the security team, key data access users, e.g. privileged accounts.
Use this CSV header line for identity information: identity,prefix,nick,first,last,suffix,email,phone,phone2,managedBy,priority,bunit,category,watchlist,startDate,endDate
IF YOU HAVE AD / LDAP ACCESS here is a sample search |ldapsearch domain=<domain_name> search="(&(objectclass=user)(!(objectClass=computer)))" |makemv userAccountControl |search userAccountControl="NORMAL_ACCOUNT"
If you are collecting Windows Active Directory information then a search like this will help validate you have added them to assets.
index=msad sourcetype=ActiveDirectory | stats count by host | rename host as asset | sort +count | join [| `assets`]
Interdependencies with other teams, systems
- Need a firewall / proxy whitelist for *.splunk.com with web proxy for software notifications and updates
- Need a firewall / proxy Whitelist rule to allow access to download the desired threatlists and APIs to enrich data, e.g. blocklists, arin.net, etc.
- Create firewall routes for assets that will forward data to the Heavy Forwarder, or Indexers
- See Splunk Ports Network connectivity - What are the Splunk ports that I need to open 
- Verify DNS is configured on the network
- Verify NTP is configured correctly with the right time on Splunk devices
- Collect SMTP server and credentials
- Collect LDAP server and credentials
- Create SSL certificates for each Splunk device
- Create a VIP and DNS entry for the search head tier.
- Create a VIP and DNS entry for the forwarding tirer, e.g. syslog, heavy forwarders
- Create a VIP and DNS entry for clustermater, deploymentserver
- Service Accounts for a Splunk System User
- Splunk Administrator Accounts, needs access to Splunk Servers (SSH or RDP)
- All Accounts need access to Splunk configurations and indexes (read/write access to filesystems)
- If Sign Sign on is desired then access to; LDAP, AD, Apache HTTP (web proxy, SSO)
- If Database connections are desired, a database service account for Splunk needs to be created.
Create a domain windows service account for splunk user Create a local user on linux for splunk Create splunk groups in the domain
Identify a SME for each technology add-on you want to deploy and feed into ES.
Develop a TA for your data sources and install on the Indexer and Enterprise Security Search Head. 
When developing a TA for ES consider these necessary field extractions: 
Review the standard logging for other device types like these:
Cisco Devices -Know which components you have installed, FWSM, ASA, PIX, ACS 
When gathering log types, consider other teams use cases. -E.g. collecting UCS logs may or may not be relevant to some security teams but is interesting to a NOC.
Installing Splunk Enterprise Security Application
If you have a support contract for Enterprise Security, then you can download the SPL file from SplunkBase. Contact your Splunk Sales team if you need access.
Before getting started take a look at known issues http://docs.splunk.com/Documentation/ES/latest/RN/KnownIssues
On the Dedicated Enterprise Security Search Head, perform the following:
- install the SPL file for the app on the SH
Install Prerequisites  Current version of Splunk Enterprise Security is 3.0 for Linux  Add it by going to to manage apps, and add the Splunk Enterprise Security App SPL file
- Install Enterprise Security App
- Add it by going to to manage apps, and add the Splunk Enterprise Security App SPL file
When you install Splunk Enterprise Security SSL will be turned on automatically. You need to install your own certs for these servers. Enable SSL/HTTPS 
- Enable SSL/HTTPS 
When you install Splunk Enterprise Security SSL will be turned on automatically. You need to install your own certs for these servers.
- Set the system-wide ui-prefs.conf to limit time range picker from All Time to Last 24 hours or Last 7 days instead of All Time.
- Configure with SMTP information for sending email alerts Setup an email alert to test SMTP 
- Setup an email alert to test SMTP 
- Outputs.conf should be deployed from the DS and set the SH to forward it’s logs the indexer
- Configure with LDAP information for users to sign on
- Create users and roles e.g. LDAP strategy
Suggested groups are Splunk Admins, ES Admins, Splunk Power Users, ES Analysts, should have mappings to some LDAP group.
- Get the Proxy information for system updates at splunk.com and various threat-lists
- Disable/ Remove unnecessary ES configuration apps.
- Disable views and saved searches for data sources that do not apply to this environment
- Change all the ES Real-time searches to Scheduled. Disable all the ones that are not in use.
- Disable demo assets and demo identities located at https://splunk:8000/en-US/manager/SplunkEnterpriseSecuritySuite/data/inputs/identity_manager
- Put the Splunk systems, and other appliances and physical systems in the static asset list with is_expected=true.
- Put the Splunk Admins, the Domain Admins in an elevated identity group, such as "high", put root /admin accounts in critical.
- Install Apps SA-*, TA-*, as needed by data sources. See Splunk Base.
You are looking for SPLUNK_ or CIM compliant add-ons for best use with ES.
################# # SEARCH HEAD - ENTERPRISE SECURITY # SPLUNK_HOME/etc/system/local/outputs.conf login_content = Enterprise Security Search Head