From Splunk Wiki
Multiple index server deployment options
Distributed search with data balancing
In this model, Splunk is installed on all servers in forwarding mode. Those forwarders balance their data output to Splunk indexes configured for distributed search. By federating the search execution across different indexes, total aggregate capability can be scaled in a linear fashion. If more performance is required, additional Splunk index servers can be brought on-line inside the distributed search group.
Splunk's data routing capabilities implement discrete data flow control to both Splunk indexes and other locations. You can implement routing rules by message content, source, sourcetype, or host to meet a wide variety of integration requirements.
Index and search tiers for massive scalability
In this model, separate physical resources are allocated to search and index. In deployments that scale beyond hundreds of gigabytes per day or have high performance requirements for both search as well as index operations, you can allocate separate resources to these operations to improve performance and achieve greater scalability.
Short-term index tier
In the short-term index tier, Splunk forwarders are deployed to all systems in the datacenter and provide IT data and change detection information to Splunk. You can then deploy a Splunk indexer to provide search capabilities for co-located operations personnel without burdening outbound network links. A deployment server instance configured on the Splunk indexer distributes configuration to the forwarders installed to systems in the datacenter. Data retention is kept within the bounds of the indexer's local disk with all data being routed to the long-term indexing tier.
Long-term index tier
In the long-term index tier, Splunk indexers are installed into the long-term index tier to aggregate data being forwarded from the short-term index tier. The index tier allows for configurations that enable the use of all system resources to maximize indexing throughput, while moving most of the data to network or off-line storage.
Systems in the search tier host the SplunkWeb user interface for the deployment's users. The Splunk servers in this tier deliver search terms to the indexing tier, and present results from the indexing tier to web users.