Community:Deploy:How To Set Up Search Head Pooling and Shared Bundle

From Splunk Wiki

Search Head pooling and Shared Bundles are are great features for Splunk distributed search deployment. Especially if you're planning to use Search Head Pooling, you should use Shared Bundles feature.

However, it is a little bit confusing when it comes to setups. Some misunderstand and think they can use the same shared directory.

They are totally different features to configure. And, the shared directories are slightly different.

• Search Head pooling is for search heads to share configurations and search jobs. And, the shared directories are "apps", "users", and "dispatch".
  
• Shared Bundle is for a search head and an indexer to share a configuration bundle. And, the shared directories are "apps", "users" and "system".
  

This example is to set up both features in the distributed search environment. Please remember that there are more than one approach to configure them.

Design Consideration

• Keep the same version over the distributed search including search peers. This is not special for search head pooling or sharing bundles. Distributed Search topology with cross-version environment has limited functionality. It's because different version has additional features or configuration parameters. Search performance might be slower. Or, you might have some warnings or errors due to cross-version.

• "splunk btool fix-dangling" after manually configuring a configuration file or using a deployment server

Under Search Head Pooling, every deletable stanza (i.e. stanza that exists purely in local) must have a corresponding ACL. Search heads interpret stanzas *without* ACLs as being "in-progress writes". They are not considered "complete" until the ACL also lands on disk and is detected. fix-dangling writes "stub" ACLs to any stanzas that lack them. There is nothing preventing puppet or other tools from deploying local.meta files with the appropriate ACLs set.
But, you must be be aware of this extra "validity", fix-dangling requirement when deploying to your search head pooling.
To be clear, fix-dangling is not really required. It doesn't do anything special. Users just need to know to modify local.meta in addition to local/*.conf or to run fix-dangling on a representative SHP instance before bundling and pushing via deployment server, puppet, etc. That said, if you're deploying only default directories, this is not required because fix-dangling is only required for stanzas that exist purely in "local".

• Do not use this nfs shared solution in WAN network. Any general network file shared systems are not for WAN network. The nfs, smba or, CIFS should not be used in WAN network. The performance is terribly slow.

• Splunk's Deployment Server to deploy apps in Search head pooling location

If you use Splunk's Deployment Server, you have to set value of "targetRepositoryLocation" in serverclass.conf to the apps directory under search head pooling location.

- serverclass.conf
[global]
targetRepositoryLocation = <search head pooling directory>/etc/apps

Or, probably you would not like to have this for all the deployment clients. In general, you should configure it in deploymentclient.conf

- deploymentclient.conf
[deployment-client]
serverRepositoryLocationPolicy = rejectAlways
repositoryLocation = /mnt/SH_pool/etc/apps

• Deployed through Deployment Server or manually configured app's configuration is not available in WebGUI

This is not a Deployment Server/Client issue, nor Search Head pooling. If you deploy a configuration file which is for system wide configuration such as email setting or authentication etc, you must edit apps/<app name>/metadata/local.meta and add export = system for the configuration or default.

- local.meta
[]
export = system

• Summary Index with SH Pooling Forward all data to indexers, instead of indexing events in each SH Pooling . Especially this is important for Summary Indexing. Summary Indexing is an action as a result of a scheduled search. In general, summary index database is in the search head where a summary-index-enabled schedule search was run. In SH Pooing topology, a scheduled search is shared by the Search Heads in the SH Pooling. One of the SHs runs a scheduled search and do summary indexing. There is no way to control to decide which search head in the pool should run a scheduled search. As a result, each Search Head in the SH Pooling will keep only partial summary indexed events. When a search head searches events in summary index, the search query only goes to search peers and its local index database, not other search heads. In order to avoid this situation, all indexing events including summary index in Search Heads in SH Pooling should be forwarded to indexers.

• System Clock Synchronization Between SHs and NFS server Make sure the system clock between SHs and NFS server are synchronized. Use NTP server to keep clock synchronization. When system clocks are off, search might be canceled due to hitting time-out of a search. Also, minimizing clock jitter/drift also cause unexpected search cancellation. Please make sure you use reliable NTP servers to synchronize system clocks.

• Bundle Replication filters Bundle Replication sent from SH to search peers contains configuration files and others from system, apps, and users directories. As default, distsearch.conf $SPLUNK_HOME/etc/system/default directory defines which files should be included in a bundle. Usually a user might need to add more files to exclude configuration or undesired large files in order to reduce the size of a bundle. You should not include configuration or any files which are already excluded as default. This might cause unexpected behavior at search time. We've seen a case a user included all of the files in a bundle accidentally by overwriting distsearch.conf.

• Upgrading SH Pooling Distributed Search topology Please check Splunk online manual for proper steps. It is very difficult to keep a Search Head running during the upgrading process. You should be prepared for maintenance downtime for a while. A backup of all the configuration and test in non-production environment are important.

• Search Performance Consideration Search Performance in SH Pooling would be slower than one in non-SH Pooling. This is simply because NFS disk access is a lot slower than local disk. So, you should consider enterprise level of dedicated storage network with SAN device. In general nfs server under shared network, we've seen terrible performance issue if pinging to nfs server from a search head takes > 2-3ms.

How To Set Up Search Head Pooling

#--------------
# SH Pooling
#--------------
#
# @NFS Server  :/home/shared/searchpooling
# @Search Head :/mnt/SH_pool
#
# Subdirectories in the directory of the SH Pooling location
#  Dirs:
#    etc/apps/<apps>
#    etc/users/<usernames>
#    var/run/splunk/dispatch/<search jobs>
#
# -server.conf
# [pooling]
# state = enabled
# storage = /mnt/SH_pool
#

1. @NFS Server: Create the Search Head pool directory and share/export it
=> NFSserver:/home/shared/searchpooling in this example

2. @Search Head: Mount the shared directory from the NFS server to the search head's mount point
=> The mount point is /mnt/SH_pool in this example
# mount -t nfs NFSServer:/home/shared/searchpooling /mnt/SH_pool

3. @Search Head: Create etc directory under the mount point /mnt/SH_pool
# mkdir /mnt/SH_pool/etc


4. @Search Head: Configure server.conf
 -server.conf
 [pooling]
 state = enabled
 storage = /mnt/SH_pool

5. @Search Head: Stop the Splunk
# splunk stop

6. @Search Head: Copy the existing apps and users directories from the Search Head's Splunk dir to the mount point
# cp -r $SPLUNK_HOME/etc/apps /mnt/SH_pool/etc/
# cp -r $SPLUNK_HOME/etc/users /mnt/SH_pool/etc/


7. @Search Head: Start the Splunk
=> This will take a minute or more in order to refer to the pool for apps, users and var/run/splunk(Splunk creates this).
# splunk start


8. @Other Search Head: Step 2, 3, 4, 5, and 7 (No Step 6)

How To Set Up Shared Bundle

#
#--------------
# Shared Bundles
#--------------
#
# @NFSServer  :/home/shared/sharedbundle
# @searchpeers:/mnt/SH_bundle
#
# Required subdirectories for the shared bundle
#  Dirs:
#    apps
#    users
#    system
#

1.@Search Head: Create "system" directory symlink to /mnt/SH_pool/etc/
ln -s $SPLUNK_HOME/etc/system /mnt/SH_pool/etc/system



2.@NFS Server: Create a share directory for Shared bundle
=> NFSServer:/home/shared/sharedbundle in this example



3.@NFS Server: Create apps  directory symlink to /home/shared/searchpooling/apps
# ln -s /home/shared/searchpooling/etc/apps /home/shared/sharedbundle/apps



4.@NFS Server: Create users directory symlink to /home/shared/searchpooling/users
# ln -s /home/shared/searchpooling/etc/users /home/shared/sharedbundle/users



5.@NFS Server: Create system directory symlink to /home/shared/searchpooling/system
# ln -s /home/shared/searchpooling/etc/system /home/shared/sharedbundle/system



6.@NFS Server: Share/Export the shared bundle directory



7.@Indexer/searchpeers: Mount the shared bundle of the NFS server to the mount point
mount -t nfs NFSServer:/home/shared/sharedbundle /mnt/SH_bundle



8. @Search Head: Disable sending the bundle
- distsearch.conf
[distributedSearch]
shareBundles = false



9. @Indexer/searchpeers: Enable the shared bundle
- distsearch.conf
[searchhead:SearchHead01-root]
mounted_bundles = true
bundles_location = /mnt/SH_bundle

=> Note that the host name is <hostname>-<splunk user> as default. It is defined as serverName in server.conf at a Search Head 
=> To find out the proper searchhead name, check "remote" jobs in the dispatch directory of the indexer, or check serverName in server.conf at the Search Head.

# cd $SPLUNK_HOME/var/run/splunk/dispatch/  
# ls | grep remote | head 5
remote_SearchHead01-root_scheduler__admin__search_VGVzdCBBQUE_at_1315712400_2f4dfd650934e0d9
remote_SearchHead01-root_scheduler__admin__search_VGVzdCBBQUE_at_1315712700_4a05440d41403c2a
remote_SearchHead01-root_scheduler__nobody__search_SW5kZXhpbmcgd29ya2xvYWQ_at_1315712700_1c5ec20a8ec47ccb
remote_SearchHead01-root_scheduler__nobody__search_VG9wIGZpdmUgc291cmNldHlwZXM_at_1315712400_69a94efac4644c64
remote_SearchHead01-root_scheduler__nobody__SplunkDeploymentMonitor_RE0gbWlzc2luZyBmb3J3YXJkZXJz_at_1315712700_ab02c550189854cb


10. @Indexer/Searchpeers: Repeat Step 9 for each search head stanza in each indexer



11. Restart the Search Heads, and then the Indexers