Splunk Architect Handbook

Start here

This book is built for the person who has to make the design calls. Not just "how do I run a search," but "how many indexers do I need, how do I lose a server without losing data, and where does this setting actually belong." If you are studying for the Splunk Enterprise Certified Architect path, the topics here line up closely with what that role is expected to know.

You can read it front to back, or jump to a topic from the sidebar. Every chapter is written to stand on its own. Code blocks show real stanza names you will type into real files. Diagrams show how the pieces connect so you can picture the whole system, not just one box.

How the book is organised

What Splunk does

Splunk takes machine data and makes it useful. Machine data is the stream of text that systems write as they run: web server logs, firewall events, application traces, operating system metrics, and much more. Splunk reads that data, stores it in a way that is fast to search, and lets people ask questions of it using a search language.

The important idea for an architect is that Splunk does not need a fixed schema up front. It stores the raw data and works out the structure at the moment you search. This is often called schema on read. It is why you can point Splunk at almost any text source and start searching within minutes, and it shapes how the whole system is built.

The three things Splunk is always doing

The data pipeline

Before any component makes sense, you need the pipeline in your head. Every piece of data that enters Splunk moves through the same set of stages. Splunk calls this the data pipeline, and the stages are input, parsing, indexing, and search. Knowing which stage a task belongs to tells you which machine does the work and which configuration file controls it.

Stage 1: Input

At the input stage Splunk acquires the raw data as a stream of bytes. It does not yet understand where one event ends and the next begins. What it does do is attach the default metadata that will travel with the data for its whole life: the source (where it came from, such as a file path), the host (the machine that produced it), and the sourcetype (the kind of data, which tells later stages how to parse it). The file that controls this stage is inputs.conf.

Stage 2: Parsing

Parsing is where the raw stream turns into individual events. Splunk decides where each event breaks, finds the timestamp for each one, and can transform the data before it is written. This is heavy work, and it is done on an indexer or on a heavy forwarder. The main files here are props.conf and transforms.conf. Line breaking, timestamp extraction, event boundaries, and any masking of sensitive fields all happen here, at index time.

Stage 3: Indexing

At indexing, the parsed events are written to disk inside an index. Splunk writes the compressed raw data and builds the index files that make time-based search fast. Data lands in a directory called a bucket. The file that defines indexes and their storage rules is indexes.conf. Once data is indexed, it is durable and ready to search.

Stage 4: Search

Search is everything that happens when a user or a scheduled job runs a query. Field extraction that was not done at index time happens now, at search time. This is the schema on read idea in action. Most field extractions, lookups, event types, and tags are applied here, which keeps them flexible because you can change them without re-indexing. The search language that drives this stage is called the Search Processing Language, or SPL.

Core components

A single Splunk Enterprise install is one piece of software. What makes it an indexer, a search head, or a forwarder is the role you give it through configuration. A small deployment might run every role on one machine. A large one spreads the roles across many machines so each can be scaled on its own. An architect needs to know every role, what it does, and how the roles talk to each other.

Forwarder

A forwarder is a Splunk instance whose job is to collect data on a machine and send it somewhere else, usually to an indexer. The universal forwarder is a small, separate download built only for this. It uses little CPU and memory and is the standard agent you install on the machines you want to monitor. A heavy forwarder is a full Splunk Enterprise instance set to forward, and it can parse and route data before sending it. Forwarders are covered in depth in the Forwarders chapter.

Indexer

An indexer receives data, parses it, indexes it, and stores it on disk. It is also what searches actually run against, so it does double duty as the storage layer and the search worker. In a distributed deployment the indexers are the heart of the system. When people talk about scaling Splunk for more data, they almost always mean adding indexers. The indexer is where your disk, your ingest capacity, and much of your search speed live.

Search head

A search head is the instance people log into to run searches, build dashboards, and manage alerts. A search head does not usually store indexed data. Instead it sends the search out to the indexers, which each run their part, and then it merges the results. This split is called distributed search. In it, the search head is the coordinator and the indexers are the search peers.

Cluster manager

The cluster manager, once called the master node, runs an indexer cluster. It does not index data itself. It keeps track of which indexers are up, decides where copies of data should live, and coordinates recovery when a node fails. It also pushes shared configuration to every indexer in the cluster so they all stay identical. There is one cluster manager per indexer cluster. It is covered in the Indexer clustering chapter.

Deployer

The deployer distributes apps and settings to the members of a search head cluster. It sits outside the cluster and cannot run on a machine that is also a cluster member. Do not confuse it with the deployment server, which manages forwarders. The deployer serves the search head cluster only. It appears again in the Search head clustering chapter.

Deployment server

The deployment server is the tool for managing configuration on many forwarders at once. You group your forwarders into server classes, map apps to those classes, and the deployment server pushes the right apps to the right machines. Without it, updating a thousand forwarders means touching a thousand machines. With it, you edit one place. See the Deployment server chapter.

License manager

The license manager, also called the license master, holds the Splunk license and tracks how much data your deployment ingests per day. Every other instance reports its daily volume to it. In a larger deployment you run one central license manager and point all other instances at it, so you have a single view of your license usage.

Monitoring Console

The Monitoring Console, sometimes still called the distributed management console, is the built-in health dashboard for the whole deployment. It shows indexing rate, search load, resource use, cluster status, and much more. In a distributed setup you dedicate one instance to run it so it can watch every other node from one place. It is your main window into whether the system is healthy.

How a single instance differs from a large deployment

Deployment tiers

There are three shapes a Splunk deployment can take, and picking the right one is one of the first jobs of an architect. The shape follows the amount of data you ingest each day, how many people search, and how much downtime you can accept. You move up the ladder only when a real need pushes you there.

Single instance

One machine does everything. It reads data, indexes it, and serves searches. This is right for a proof of concept, a lab, a small team, or a department with light data. It is simple to run because there is only one box to watch. The limit is obvious: when that one machine runs out of ingest or search capacity, or when you cannot afford it going down, you have to grow.

Distributed, non-clustered

You separate the roles. One or more indexers store and search the data, and one or more search heads serve the users. Forwarders feed the indexers. This spreads the load so each tier scales on its own. You can add indexers to handle more data and add search heads to handle more users. What it does not give you on its own is automatic recovery. If an indexer dies, the data only on that indexer is unavailable until you fix it.

Clustered

This is the highly available shape. The indexers form an indexer cluster that keeps multiple copies of the data across nodes, so losing a node does not lose data or stop searches. The search heads form a search head cluster that keeps user content in sync and shares the search load. This is what you build when data cannot be lost and the service must stay up. It costs more hardware and more operational care, which is the trade you make for resilience.

Splunk Validated Architectures

You do not have to invent your topology from nothing. Splunk publishes a set of tested reference designs called Splunk Validated Architectures, or SVAs. These are approved patterns for common needs, from a single server up to a multi-site cluster. Starting from an SVA means you begin with a shape Splunk has already proven, then adjust it to your case.

How to read an SVA

Each validated architecture describes the tiers involved, how many of each component you need, and what problem it solves. They are labeled with short codes. The idea is to first pick the design that matches your availability and scale needs, then size the hardware inside it. The pattern gives you the shape, and the sizing work gives you the numbers.

The pillars behind the patterns

Splunk frames good design around a few qualities: availability so the service stays up, performance so searches are fast, scalability so you can grow, security so data is protected, manageability so you can run it, and cost so it stays affordable. A validated architecture is a balance point across these. When you change a design, check what you are trading. Removing a search head cluster saves money but costs availability. Adding a site adds resilience but adds complexity.

Forwarders

Forwarders are how data gets into Splunk from the machines you want to watch. You install a small agent on each source machine, and it reads the logs and sends them onward. Getting this tier right matters, because it is usually the largest part of a deployment by machine count. You may have a handful of indexers and thousands of forwarders.

Universal forwarder versus heavy forwarder

How forwarders find indexers

A forwarder sends data to a set of receivers. The cleanest way to configure this is with an indexer discovery setup in a cluster, where the forwarder asks the cluster manager for the current list of peers. Without discovery, you list the receivers directly. The key file on the forwarder is outputs.conf, and the receiving side turns on a listener in inputs.conf.

outputs.conf on the forwarder

# Send to a group of indexers and spread the load
[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = idx1.example.com:9997, idx2.example.com:9997, idx3.example.com:9997

# useACK makes the forwarder wait for the indexer to confirm receipt
useACK = true

inputs.conf on the receiving indexer

# Open the receiving port so forwarders can connect
[splunktcp://9997]
disabled = 0

Indexer acknowledgement

By default a forwarder considers data sent once it hands it to the network. Turn on useACK and the forwarder holds each block until the indexer confirms it was written. This protects against losing data if an indexer or the network drops at the wrong moment. It costs a little memory on the forwarder for the wait queue. For data you cannot afford to lose, turn it on.

Intermediate forwarders

Sometimes you place a forwarding tier between the source forwarders and the indexers. This is an intermediate forwarder, and it is used to cross network boundaries, to funnel traffic through a firewall gap, or to gather data from many small forwarders before sending it on. Use them when the network shape requires it, but know that every extra hop is another thing to run and can become a bottleneck if it is under-sized.

Deployment server

Once you have more than a few forwarders, you cannot manage them by hand. The deployment server solves this. It is a Splunk instance that pushes configuration, in the form of apps, to a fleet of forwarders. You define groups of forwarders, decide which apps each group should get, and the deployment server keeps them in sync. Change one file in one place, and every matching forwarder picks it up.

The pieces

How a client checks in

On each forwarder you point it at the deployment server with deploymentclient.conf. Giving each client a friendly name with clientName makes it much easier to target later.

# deploymentclient.conf on the forwarder
[deployment-client]
clientName = web_tier_linux

[target-broker:deploymentServer]
targetUri = deployserver.example.com:8089

Mapping clients to apps

On the deployment server, serverclass.conf defines which clients belong to which class and which apps that class receives. You match clients by hostname, IP, DNS name, or the client name you set above.

# serverclass.conf on the deployment server
[serverClass:web_tier]
whitelist.0 = web_tier_linux

[serverClass:web_tier:app:Splunk_TA_nix]
restartSplunkd = true
stateOnClient = enabled

Scale limits

A single deployment server can manage a large number of clients, but there is a practical ceiling that depends on how often clients check in and how big the apps are. For very large fleets you increase the check-in interval, keep apps small, or run more than one deployment server. Watch the deployment server's own load through the Monitoring Console, and confirm the current supported client count against the docs for your build before you plan a very large fleet.

Inputs and parsing

Getting data in well is where good and bad deployments separate. Two settings decide the quality of everything downstream: the sourcetype you assign, and the parsing rules attached to it. Get these right at the start and searches are fast and clean. Get them wrong and you pay for it on every search forever.

Common input types

• Files and directories. Splunk tails log files and picks up new lines as they are written. The most common input by far.
• Network inputs. Splunk listens on a TCP or UDP port for data such as syslog. TCP is preferred over UDP because UDP can silently drop data under load.
• Scripted inputs. A script runs on a schedule and its output becomes data. Useful for pulling from a command or a small API.
• Modular inputs and HTTP Event Collector. Structured ways for apps and code to send data. HTTP Event Collector, or HEC, lets applications send events straight to Splunk over HTTP with a token, with no forwarder needed.

The big eight parsing settings

Most parsing problems come down to a small set of settings in props.conf, applied per sourcetype. Setting these explicitly, rather than letting Splunk guess, makes ingest faster and more reliable.

A clean sourcetype definition

# props.conf, applied on the indexer or heavy forwarder
[my_app_log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 25
TZ = UTC
TRUNCATE = 10000

Indexer clustering

An indexer cluster is a group of indexers that keep copies of each other's data. If one indexer fails, the copies on the others keep the data available and searchable. This is how Splunk delivers data availability. Understanding the cluster is the single most important skill for an architect, because it is where your data safety and much of your capacity planning live.

The roles

• Cluster manager. One per cluster. It coordinates the peers, tracks the copies, and drives recovery. It does not index data.
• Peer nodes. The indexers themselves. They receive data, index it, keep copies of other peers' data, and run searches.
• Search heads. They get their list of peers from the cluster manager and search across all of them.

Replication factor and search factor

Two numbers define how safe your data is. The replication factor is how many total copies of each bucket of data the cluster keeps. The search factor is how many of those copies are searchable, meaning they include the index files needed to search, not just the raw data.

The common default is a replication factor of 3 and a search factor of 2. That means each bucket exists on three peers, and two of those copies are fully searchable. With this setup the cluster can lose one peer and keep all data available and searchable. The trade is disk: a replication factor of 3 means you store roughly three times the raw indexed volume, and the searchable copies add more on top.

The bucket lifecycle

Data does not sit still. Each bucket moves through stages as it ages, from hot where new data is written, to warm, to cold, and finally to frozen where it leaves the searchable system. This lifecycle is how Splunk balances fast recent data against cheap old storage. It is covered in detail in the Indexes and buckets chapter, and it applies inside a cluster just as it does on a single indexer.

Setting up the manager

# server.conf on the cluster manager
[clustering]
mode = manager
replication_factor = 3
search_factor = 2
pass4SymmKey = yourClusterSecret
cluster_label = prod_cluster

server.conf on each peer

[clustering]
mode = peer
manager_uri = https://cm.example.com:8089
pass4SymmKey = yourClusterSecret
[replication_port://9887]

Multi-site clusters

A multi-site cluster spreads peers across two or more locations and lets you say how many copies must live at each site. This protects you from losing a whole site, not just a single server. It uses site-aware replication and search so that, where possible, a search reads from local copies to save bandwidth. Multi-site adds real complexity, so use it only when a site failure is a risk you must plan for.

Search head clustering

A search head cluster is a group of search heads that share the same user content and spread the search workload. If one member fails, users keep working on the others, and their dashboards, saved searches, and reports are all still there. This is how Splunk delivers availability for the search tier.

The captain

One member of the cluster acts as the captain. The captain is not a separate machine. It is a role that one of the members takes on. The captain coordinates the cluster: it schedules the shared jobs, replicates changes to knowledge objects across all members, and keeps everyone in sync. The role is dynamic, meaning any member can become captain, and the cluster elects a new one automatically if the current captain fails.

The deployer

The deployer is a separate instance that pushes apps and base configuration to all the cluster members. It must not be a member itself, and it must not run on a cluster member. You use it to roll out an app or a settings change to every member in one action. Day-to-day user content, such as a dashboard someone saves, is handled by replication among the members, not by the deployer. The deployer is for the shared base; replication is for live user changes.

Why you need an odd number of members

The captain is chosen by a majority vote. For a vote to have a clear majority, you want an odd number of members. With an odd count the cluster can always form a majority on one side of a network split, which prevents two halves from both electing a captain. The common minimum for a real search head cluster is three members. Fewer than three cannot safely handle the loss of a member and still hold a majority.

Connecting it to the indexers

To search clustered indexers, you configure each search head cluster member as a search head on the indexer cluster. Then the members get their peer list from the cluster manager automatically. Splunk recommends that the secret key for the search head cluster be different from the secret key for the indexer cluster, so the two clusters stay cleanly separated.

Indexes and buckets

An index is where Splunk stores data on disk. Inside an index, data is held in directories called buckets, and each bucket holds events from a span of time. Understanding buckets is how you understand storage cost, search speed, and retention. It is one of the most practical skills for an architect, because it turns "keep 90 days of data" into real disk numbers.

What is in a bucket

A bucket holds two main things: the compressed raw data as it arrived, and the index files that make searching fast. The raw data compresses well, often to a small fraction of its original size. The index files add some overhead on top. The combined size on disk is what you plan capacity around, not the raw log size alone.

The bucket lifecycle

Buckets age through a fixed set of stages. New data is written to hot buckets. When a hot bucket fills up or reaches an age limit, it rolls to warm, meaning it is closed but still on fast storage. Later it rolls to cold, which can be on cheaper, slower storage. Finally it rolls to frozen, at which point Splunk either deletes it or hands it to you to archive. There is also a thawed state for data you bring back from an archive.

Controlling retention and storage

You control all of this in indexes.conf. The two levers that matter most are time and size. Data freezes when it is older than frozenTimePeriodInSecs, or when the index grows past maxTotalDataSizeMB, whichever comes first. You point hot and warm at fast storage with homePath, cold at cheaper storage with coldPath, and set where frozen data goes.

# indexes.conf
[web_prod]
homePath   = /fast/splunk/web_prod/db
coldPath   = /bulk/splunk/web_prod/colddb
thawedPath = /bulk/splunk/web_prod/thaweddb

# keep data for 90 days
frozenTimePeriodInSecs = 7776000

# cap the index at 500 GB
maxTotalDataSizeMB = 500000

# run this script when a bucket freezes, instead of deleting it
coldToFrozenScript = /opt/splunk/bin/archive_bucket.sh

Index design tips

• Separate indexes by data type and access needs. Put data with different retention, sensitivity, or access rules in different indexes. Access control is granted per index, so this is how you keep the right eyes on the right data.
• Do not make hundreds of tiny indexes. Every index carries some overhead. Group sensibly rather than creating one per source.
• Plan retention up front. Retention drives disk, and disk drives cost. Know how long each index must hold data before you size anything.
• In a cluster, indexes.conf must match on every peer. Push it from the cluster manager so all peers agree.

SmartStore

SmartStore changes where the bulk of your data lives. Instead of keeping all warm and cold data on local disk on each indexer, SmartStore keeps the master copy in remote object storage, such as Amazon S3 or a compatible on-premises store, and uses local disk only as a cache for the data being actively searched. This lets you scale storage and compute separately, which is a major shift in how you size a deployment.

Why it matters for design

Without SmartStore, if you need to keep more data, you add indexers just to get more local disk, even if you do not need more compute. With SmartStore, you grow cheap object storage for capacity and add indexers only when you need more compute or ingest. This decoupling is the whole point.

Sizing the local cache

The local disk becomes a cache, not the full store. It must be large enough to hold the working set, meaning the data your searches actually touch. Splunk guidance is to size the cache to hold roughly the equivalent of the most recent period your searches commonly cover, often described as around 30 days of data, though the right number depends on your search patterns. If searches often reach older data, that data has to be fetched from the remote store, which is slower.

Configuration files

Almost everything in Splunk is controlled by plain text files that end in .conf. The web interface is a friendly front end, but underneath it writes to these same files. An architect works with the files directly, because that is where the real control and the real troubleshooting happen. Learn the file system and you can configure and fix anything.

How the files are organised

Configuration lives inside apps. An app is just a folder under $SPLUNK_HOME/etc/apps. Inside each app, settings can live in three places, and the place decides the priority:

• default. The default folder holds settings that ship with the app. You never edit these, because an upgrade can overwrite them.
• local. The local folder holds your changes. This is where you put your settings, and it wins over default.
• System. The etc/system/local folder holds deployment-wide settings that sit above the apps.

The structure of a .conf file

Every conf file has the same shape. Square brackets define a stanza, which is a named block. Under each stanza are settings written as key = value. Comments start with a hash.

# a stanza header in square brackets
[my_sourcetype]
# key = value pairs under it
SHOULD_LINEMERGE = false
TItle = example

Precedence and btool

When the same setting appears in more than one place, Splunk has to decide which one wins. That decision is called configuration file precedence, and it is one of the most important things an architect must truly understand. Most "why is this setting not working" problems are really precedence problems: your setting exists, but another copy with higher priority is overriding it.

The order that decides the winner

For most files, when Splunk merges settings it takes the highest priority copy first, then fills in anything missing from lower priority copies. The general order, from highest to lowest, is system local, then app local, then app default, then system default. In short, your system-wide overrides beat app overrides, and any local beats any default.

btool, the settings you actually get

Because precedence is layered, the setting you wrote is not always the setting that wins. The btool command shows you the final merged result, taking precedence into account. It is your ground truth for "what will Splunk actually use." It reports what would apply on the next restart, in order of precedence.

# show the merged props.conf as Splunk will use it
$SPLUNK_HOME/bin/splunk btool props list --debug

# check one sourcetype and see which file each setting came from
$SPLUNK_HOME/bin/splunk btool props list my_sourcetype --debug

# check inputs, outputs, indexes the same way
$SPLUNK_HOME/bin/splunk btool inputs list --debug

Key .conf reference

There are many conf files, but a working architect touches a core set constantly. Here is what each one does and where it belongs. Knowing the right file for a job is half the battle.

Which tier needs which file

A frequent mistake is putting a setting on the wrong tier. Parsing settings in props.conf need to be where parsing happens, which is the indexers or heavy forwarders. Search-time field extractions in props.conf need to be on the search heads. Receiving ports in inputs.conf belong on the indexers. Sending settings in outputs.conf belong on the forwarders. When a setting does not seem to work, first ask whether it is even on the right machine.

Sizing and capacity

Sizing is the work of turning a requirement into a number of machines. It is where an architect earns their keep. Under-size and the system is slow and fragile. Over-size and you waste money. The good news is that sizing follows a clear method: start from ingest and search load, apply the reference hardware, and add up.

Reference hardware

Splunk publishes reference hardware specifications so you have a known-good baseline for a single indexer or search head. For the 9.4 line, Splunk describes tiers of specification. The mid-range specification is built around 24 physical CPU cores, or 48 vCPU at 2 GHz or greater per core, along with generous memory and fast storage. There is a smaller entry specification and a high-performance specification above it.

The storage side

Indexer storage must be fast. Splunk expresses this in IOPS, the number of input and output operations per second the disks can sustain. As a scale example from Splunk's own guidance, a shared storage array serving ten indexers at solid-state performance would need on the order of 40,000 concurrent IOPS, which is roughly 4,000 IOPS per indexer. Slow storage starves searches no matter how much CPU you have, so this is not a place to cut corners.

How many indexers

The main driver of indexer count is daily ingest volume, balanced against search load. Each indexer can handle a certain amount of data ingest per day at the reference specification, and searching adds to that load. You divide your total daily volume by the per-indexer ingest capacity, then add indexers for search concurrency, for headroom, and for the replication overhead if you cluster.

A worked way of thinking

1. Find your daily ingest in gigabytes per day, and be honest about peaks, not just averages.
2. Divide by the ingest each indexer can handle at your chosen specification to get a base indexer count.
3. Add indexers for search load, since heavy searching reduces how much each indexer can ingest.
4. If clustering, remember replication multiplies the disk you need and adds some CPU.
5. Add headroom so you are not running at the edge, and so you can lose a node without pain.

Search head sizing

Search heads are driven by how many people search, how many scheduled searches run, and how heavy those searches are. The reference hardware applies here too. In a search head cluster you add members for both availability and to spread scheduled search load, keeping an odd number so captain election stays clean.

Security

A Splunk deployment holds a lot of sensitive data, and it is often part of the security team's own tooling, so it has to be locked down well. Security in Splunk falls into a few clear areas: who can log in, what they can do once in, and how data is protected as it moves and rests.

Authentication

Authentication is how users prove who they are. Splunk supports local accounts, but in any real deployment you tie it to your existing identity system, usually LDAP or Active Directory, or single sign-on through SAML. This is set in authentication.conf. Central identity means you manage joiners and leavers in one place, and you get your organisation's password and multi-factor rules for free.

Authorization with roles

Authorization is what a user is allowed to do. Splunk uses role-based access control. You assign users to roles, and each role carries a set of capabilities and a set of indexes it can search. This is defined in authorize.conf. Because index access is granted per role, splitting sensitive data into its own index is how you keep it visible only to the right roles.

You rarely rely on the built-in roles alone. You build custom roles that match your teams, grant each only the indexes and capabilities it needs, and inherit from the built-ins where it helps. Least privilege is the goal: give each role the smallest set of powers that lets it do its job.

Protecting data in motion and at rest

• Encrypt traffic. Turn on TLS for forwarder-to-indexer traffic, for the management port, and for the web interface. This protects data as it crosses the network.
• Mask sensitive data at index time. If data contains secrets you must not store, use props.conf and transforms.conf to mask or drop it before it is written. Once it is indexed, it is much harder to remove.
• Protect the secret keys. The cluster secret keys, set with pass4SymmKey, guard membership in your clusters. Keep them safe and use different keys for the indexer cluster and the search head cluster.
• Control the management port. The management port, usually 8089, is powerful. Restrict who can reach it at the network level.

Monitoring and troubleshooting

A large Splunk deployment needs to be watched like any other critical system. The good news is that Splunk is very good at watching itself, because it indexes its own logs. Your main tools are the Monitoring Console, Splunk's internal logs, and a handful of commands.

The Monitoring Console

The Monitoring Console is the built-in health center for the whole deployment. It shows indexing rate, search activity, resource use per instance, cluster health, and much more. In a distributed setup you set up one instance as the console and register every other node with it, so you get a single pane of glass across the environment. Check it regularly, and set up its health alerts so problems find you instead of the other way around.

Splunk's own logs

Splunk writes its internal logs into an index called _internal. Because it is just another index, you can search it with SPL like any data. This is where you go when something is wrong. The most useful file is splunkd.log, which records what the core process is doing, including errors and warnings.

# find recent errors across the deployment
index=_internal source=*splunkd.log log_level=ERROR

# watch indexing throughput per indexer
index=_internal source=*metrics.log group=per_index_thruput

Common problems and where to look

Useful command line tools

# overall health of an indexer cluster, run on the cluster manager
splunk show cluster-status

# search head cluster status, run on a member
splunk show shcluster-status

# validate configuration files before a restart
splunk btool check

Operations Book

This chapter is the hands-on companion to the rest of the handbook. It is the page you keep open while you work. It gathers the directories and configuration files you touch most, the commands to run and manage Splunk day to day, the commands to add data and monitors, and clear guidance on which machine each command belongs on. Everything here is meant to be copied and used.

Directories and conf files

This is the master table. It lists the important directories first, then the configuration files grouped by what they do. Every path is relative to $SPLUNK_HOME unless stated. Use it to answer two questions fast: where does this thing live, and which file controls this behavior.

Important directories

Configuration files by purpose

These are the files you place inside an app's local folder, or in etc/system/local. The tier column tells you which machine the file usually belongs on.

Getting data in

Storage and indexing

Instance, roles, and clustering

Security and access

Search-time knowledge

Start, stop, and manage

These are the commands that run the Splunk process itself. On Linux, change into $SPLUNK_HOME/bin and run them as ./splunk, or use the full path /opt/splunk/bin/splunk. On Windows, run splunk from the same bin folder in a command prompt, or use the Services panel. Many commands need administrator or root rights, so you may need sudo on Linux.

Service control

# from $SPLUNK_HOME/bin
./splunk start                 # start Splunk
./splunk stop                  # stop Splunk
./splunk restart               # stop then start
./splunk status                # is it running
./splunk version               # show the version

# accept the license without the prompt, useful in scripts
./splunk start --accept-license

# restart only one part instead of the whole instance
./splunk restart splunkd       # the core engine
./splunk restart splunkweb     # the web interface

Start at boot

# make Splunk start automatically when the machine boots
sudo ./splunk enable boot-start

# run it as a systemd service (modern Linux), managed by systemd
sudo ./splunk enable boot-start -systemd-managed 1 -user splunk

# turn boot start off
sudo ./splunk disable boot-start

Config, users, and indexes

# check that config files are valid before a restart
./splunk btool check

# see the final merged settings for a file, with source files
./splunk btool inputs list --debug

# change the admin password
./splunk edit user admin -password 'NewStrongPass' -auth admin:oldpass

# add and list indexes
./splunk add index security_logs
./splunk list index

# reload the deployment server after editing serverclass.conf
./splunk reload deploy-server

# log in once so later commands do not ask for credentials
./splunk login -auth admin:yourpassword

Cluster and search head cluster

# run on the cluster manager: push shared config to all indexer peers
./splunk apply cluster-bundle

# run on the cluster manager: overall indexer cluster health
./splunk show cluster-status

# run on a search head cluster member: cluster status
./splunk show shcluster-status

# run on a search head: add a standalone indexer as a search peer
./splunk add search-server https://idx1.example.com:8089 -auth admin:pass -remoteUsername admin -remotePassword pass

Add data and monitors

This is how you tell Splunk to start reading a source from the command line, without editing files by hand. The general pattern is ./splunk [add|edit|remove|list] [monitor|oneshot|tcp|udp] source -parameter value. These commands write to inputs.conf for you. Run them on the machine that will read the data, which is usually a forwarder.

Monitor files and directories

# watch a single log file, picking up new lines as they arrive
./splunk add monitor /var/log/messages

# watch a whole directory and everything under it
./splunk add monitor /var/log/nginx/

# set the index and sourcetype at the same time
./splunk add monitor /var/log/app/app.log -index app_logs -sourcetype my_app_log

# override the host value for this input
./splunk add monitor /var/log/dmesg -hostname web01 -index os_logs

# list, edit, and remove monitors
./splunk list monitor
./splunk edit monitor /var/log/app/app.log -index new_index
./splunk remove monitor /var/log/app/app.log

Index a file one time

# read a file once, right now, and do not keep watching it
./splunk add oneshot /var/log/old/archive.log -index import_index -sourcetype my_app_log

Network inputs

# listen for data on a TCP port, for example raw syslog
./splunk add tcp 514 -sourcetype syslog -index network

# listen on a UDP port (prefer TCP where you can, UDP can drop data)
./splunk add udp 514 -sourcetype syslog -index network

Forwarding and receiving

# on a forwarder: send data to an indexer
./splunk add forward-server idx1.example.com:9997

# on the receiving indexer: turn on the listener for forwarded data
./splunk enable listen 9997

# list and remove forward destinations
./splunk list forward-server
./splunk remove forward-server idx1.example.com:9997

Where to run commands

The most common cause of a command not working as expected is running it on the wrong machine. Splunk commands act on the local instance, so you must run each one on the instance that owns that job. This table maps the common commands to where they belong.

Q&A: how-to guides

This is the practical part of the book. Each guide below takes one real task an architect gets asked to do and walks through it step by step, with the exact files, commands, and the machine each step runs on. Read the ones you need, or work through them all. For quick recall practice, use the separate Quick-fire Q&A page, and to test yourself, use the Exercises page.

1. Filter and route events using a Splunk forwarder

Goal: drop the events you do not want, and send the rest to different indexers or indexes based on their content. Because this needs the data to be parsed first, it must run on a heavy forwarder or on the indexers, not on a universal forwarder. A universal forwarder does not parse, so it cannot filter by content.

Step 1. Understand the two jobs

Filtering means sending unwanted events to a special destination called nullQueue, which throws them away. Routing means tagging events with a target group so they go to a chosen set of receivers. Both are done with a pair of files: props.conf assigns a transform to a sourcetype, and transforms.conf defines what that transform does.

Step 2. Filter: drop events you do not want

Say you want to keep everything except the noisy DEBUG lines from a sourcetype called app_log. On the heavy forwarder or indexer, in an app's local folder:

# transforms.conf
# first, a catch-all could route everything, then this sends DEBUG to the void
[drop_debug]
REGEX = \bDEBUG\b
DEST_KEY = queue
FORMAT = nullQueue

# props.conf
[app_log]
TRANSFORMS-drop = drop_debug

# transforms.conf, keep only lines containing ERROR or WARN
[set_null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keep_important]
REGEX = (ERROR|WARN)
DEST_KEY = queue
FORMAT = indexQueue

# props.conf, apply both in order
[app_log]
TRANSFORMS-filter = set_null, keep_important

Step 3. Route: send events to a specific indexer group

Define the target groups in outputs.conf, then use a transform to stamp matching events with the group name via the special key _TCP_ROUTING. On the heavy forwarder:

# outputs.conf, two named groups
[tcpout]
defaultGroup = primary

[tcpout:primary]
server = idxA:9997, idxB:9997

[tcpout:security_idx]
server = sec-idx1:9997, sec-idx2:9997

# transforms.conf, send firewall events to the security group
[route_firewall]
REGEX = firewall
DEST_KEY = _TCP_ROUTING
FORMAT = security_idx

# props.conf
[app_log]
TRANSFORMS-route = route_firewall

Step 4. Apply and confirm

./splunk btool props list app_log --debug   # confirm the transforms are attached
./splunk restart                            # or reload as appropriate

2. Monitor Windows logs using a universal forwarder

Goal: collect Windows Event Logs, such as Security and System, from a Windows host and send them to your indexers. This runs on the Windows universal forwarder installed on that host.

Step 1. Install the universal forwarder on Windows

Install the Windows universal forwarder package on the host. During install you can point it at your deployment server and your receiving indexers, or set those later. The forwarder runs as a Windows service called SplunkForwarder.

Step 2. Define the Windows inputs

Windows Event Logs use a special input stanza. On the forwarder, in %SPLUNK_HOME%\etc\apps\your_app\local\inputs.conf:

# inputs.conf on the Windows universal forwarder
[WinEventLog://Security]
disabled = 0
index = win_security

[WinEventLog://System]
disabled = 0
index = win_os

[WinEventLog://Application]
disabled = 0
index = win_os

Step 3. Point the forwarder at your indexers

# outputs.conf on the forwarder
[tcpout]
defaultGroup = indexers

[tcpout:indexers]
server = idx1:9997, idx2:9997

Or set it from the command line on the forwarder:

splunk add forward-server idx1:9997 -auth admin:password

Step 4. Make sure the index exists and restart

On the indexers (or the cluster manager for a cluster), create the target indexes so the data has a home, then restart the forwarder.

# on an indexer
splunk add index win_security
splunk add index win_os

# on the Windows forwarder
splunk restart

Step 5. Confirm data is arriving

On a search head, run a search over a recent window: index=win_security. If nothing shows, check the receiving port is enabled on the indexer, and read splunkd.log on the forwarder for connection errors.

3. How does Splunk scale?

Goal: understand the levers you pull to make Splunk handle more data and more users. The short answer is that Splunk scales out, not up. You add more machines to each tier rather than buying one giant server.

Scaling ingest and storage

Ingest and storage live on the indexers. To handle more data per day, you add more indexers and spread the incoming data across them. Forwarders load balance automatically across all the indexers you give them, so each new indexer takes a share of the load. More indexers also means each search touches a smaller slice of data, so searches stay fast as data grows.

Scaling search and users

Search and users live on the search heads. To handle more people and more scheduled searches, you add search heads. When you need availability as well, you group them into a search head cluster so user content stays in sync and the load spreads across members.

Scaling availability

To survive a machine failing, you cluster. An indexer cluster keeps multiple copies of data across peers so losing a peer loses no data. A search head cluster keeps user content on every member so losing a member loses no work. A multi-site cluster spreads copies across locations to survive a whole site failing.

Scaling storage separately from compute

With SmartStore you keep the bulk of data in cheap remote object storage and use local disk only as a cache. This lets you grow storage without adding indexers just for disk, so you scale storage and compute on their own schedules.

4. Distributed search setup and configuration

Goal: let one search head search across several separate indexers. This is the simplest distributed setup, without clustering. You configure the search peers on the search head.

Step 1. Prepare the indexers as search peers

Each indexer that will be searched is a search peer. It just needs to be running and reachable on its management port, usually 8089, from the search head.

Step 2. Add each peer on the search head

On the search head, add every indexer as a search peer. Run this once per peer:

./splunk add search-server https://idx1.example.com:8089 \
    -auth admin:shpassword \
    -remoteUsername admin -remotePassword idxpassword

./splunk add search-server https://idx2.example.com:8089 \
    -auth admin:shpassword \
    -remoteUsername admin -remotePassword idxpassword

This writes to distsearch.conf and exchanges keys so the search head can query each peer.

Step 3. Confirm the peers

./splunk list search-server

Or check Settings, then Distributed search, then Search peers in the web interface. Peers should show as Up.

Step 4. Search

Run any search on the search head. It now fans out to all peers and merges the results. No special syntax is needed. If a peer shows down, check the management port is reachable and the credentials are correct.

5. Configure the deployer and set up a search head cluster

Goal: build a working search head cluster of three members, with a deployer that pushes apps to them. This touches four machines: the deployer and three members. A search head cluster needs at least three members for safe captain election.

Step 1. Set up the deployer

The deployer is a standalone Splunk instance that is not a member. On the deployer, edit server.conf to set a shared secret and a cluster label. The pass4SymmKey here must match the one you give the members.

# server.conf on the deployer
[shclustering]
pass4SymmKey = yourShcSecret
shcluster_label = prod_shc

Restart the deployer after this change.

Step 2. Initialise each member

On each of the three members, run the init command. Use the same secret and point each member at the deployer. Do not run this on the deployer.

./splunk init shcluster-config \
    -auth admin:password \
    -mgmt_uri https://sh1.example.com:8089 \
    -replication_port 9200 \
    -replication_factor 3 \
    -conf_deploy_fetch_url https://deployer.example.com:8089 \
    -secret yourShcSecret \
    -shcluster_label prod_shc

./splunk restart

Change -mgmt_uri to each member's own address as you run it on that member. The replication factor here is how many copies of user content the cluster keeps across members.

Step 3. Bootstrap the captain

Run the bootstrap command on one member only. List all members in the servers list. This starts the cluster and elects the first captain.

./splunk bootstrap shcluster-captain \
    -servers_list "https://sh1.example.com:8089,https://sh2.example.com:8089,https://sh3.example.com:8089" \
    -auth admin:password

Step 4. Confirm the cluster is healthy

./splunk show shcluster-status -auth admin:password

You should see all three members and one captain.

Step 5. Push apps from the deployer

Place apps in the deployer's $SPLUNK_HOME/etc/shcluster/apps folder, then push them to all members with one command on the deployer:

./splunk apply shcluster-bundle -target https://sh1.example.com:8089 -auth admin:password

6. Add or remove members from a search head cluster

Goal: grow or shrink an existing cluster safely. These commands run on the member being added, or on any current member for a removal.

Add a member

First initialise the new instance with the same settings as the others, using init shcluster-config as in guide 5, then restart it. Then join it to the cluster. You can run the add command on the new member, pointing at a current member:

# on the NEW member, point at any existing member
./splunk add shcluster-member \
    -current_member_uri https://sh1.example.com:8089 \
    -auth admin:password

Or run it from a current member, pointing at the new one:

# on a CURRENT member, point at the new member
./splunk add shcluster-member \
    -new_member_uri https://sh4.example.com:8089 \
    -auth admin:password

Remove a member

Run the remove command on any cluster member. For a member that is still running, target it by its URI:

./splunk remove shcluster-member -mgmt_uri https://sh4.example.com:8089 -auth admin:password

If you plan to keep that instance for another use, disable its cluster config after removing it, and do not stop it before you remove it. For a member that has already failed and is down, run the same command from a healthy member to clean it out of the cluster.

7. Configure a static captain in a search head cluster

Goal: pin the captain role to one specific member instead of letting the cluster elect one. You do this in a special situation: when the cluster has lost so many members that it cannot form a majority to elect a dynamic captain, such as during a site outage. A static captain lets the survivors keep working. These commands run on every remaining member.

Step 1. Pick the member to be captain

Choose one healthy member. On that member, set it to be the static captain and turn off election:

./splunk edit shcluster-config -election false -mode captain \
    -captain_uri https://sh1.example.com:8089 -auth admin:password

Step 2. Point every other member at that captain

On each remaining member, turn off election and name the same captain:

./splunk edit shcluster-config -election false -mode member \
    -captain_uri https://sh1.example.com:8089 -auth admin:password

Step 3. Confirm

./splunk show shcluster-status -auth admin:password

The chosen member should now hold the captain role, and election should be off.

Step 4. Return to a dynamic captain when recovered

Once enough members are back to form a majority, turn election back on so the cluster manages the captain itself again. On every member:

./splunk edit shcluster-config -election true -mgmt_uri https://shX.example.com:8089 -auth admin:password
./splunk restart

8. Set up an indexer cluster (bonus)

Goal: build a basic single-site indexer cluster with one manager and several peers. Runs on the cluster manager and each peer.

On the cluster manager

# server.conf on the manager
[clustering]
mode = manager
replication_factor = 3
search_factor = 2
pass4SymmKey = yourClusterSecret
cluster_label = prod_idx

On each peer

# server.conf on each peer
[clustering]
mode = peer
manager_uri = https://cm.example.com:8089
pass4SymmKey = yourClusterSecret
[replication_port://9887]

Restart the manager first, then each peer. On a search head, connect to the cluster so it learns the peer list:

# server.conf on the search head
[clustering]
mode = searchhead
manager_uri = https://cm.example.com:8089
pass4SymmKey = yourClusterSecret

Push shared config to all peers from the manager with ./splunk apply cluster-bundle, and check health with ./splunk show cluster-status.

9. Mask sensitive data before indexing (bonus)

Goal: hide secrets, such as card numbers, so they are never stored. This is index-time work, so it runs on the indexers or a heavy forwarder. The simplest tool is SEDCMD in props.conf.

# props.conf, replace all but the last 4 digits of a 16 digit number
[app_log]
SEDCMD-mask_card = s/\d{12}(\d{4})/xxxxxxxxxxxx\1/g

10. Manage forwarders with a deployment server (bonus)

Goal: push an app to many forwarders from one place. Runs on the deployment server and each forwarder.

On each forwarder

# deploymentclient.conf
[deployment-client]
clientName = linux_web

[target-broker:deploymentServer]
targetUri = ds.example.com:8089

Or set it from the command line: ./splunk set deploy-poll ds.example.com:8089, then restart.

On the deployment server

Put the app in $SPLUNK_HOME/etc/deployment-apps/, define who gets it in serverclass.conf, then reload:

[serverClass:web]
whitelist.0 = linux_web
[serverClass:web:app:Splunk_TA_nix]
restartSplunkd = true
stateOnClient = enabled

./splunk reload deploy-server

Best practices

These are the habits that separate a deployment that runs quietly for years from one that fights you every week. None of them are complicated. They are about discipline and doing the boring things consistently.

Configuration

• Put every change in local, never in default, so upgrades do not wipe your work.
• Package related settings into named apps rather than scattering them. Apps are easy to move, version, and reason about.
• Use btool whenever a setting does not behave. Do not guess about precedence.
• Manage forwarders through the deployment server, cluster peers through the cluster manager, and search head members through the deployer. One tool per tier.

Getting data in

• Default to the universal forwarder. Use heavy forwarders only for a real reason such as API inputs or heavy filtering.
• Set the sourcetype and parsing settings explicitly. Do not rely on Splunk guessing line breaks and timestamps.
• Put index-time parsing settings on the parsing tier, meaning indexers or heavy forwarders, not on universal forwarders.
• Turn on indexer acknowledgement for data you cannot afford to lose.
• Land syslog on a dedicated collector and read the files with a forwarder, rather than pointing syslog straight at a Splunk UDP port.

Storage and indexes

• Decide retention and archive policy before data flows, and set the freeze behavior so you do not silently delete data.
• Separate data into indexes by retention, sensitivity, and audience, because access is granted per index.
• Keep hot and warm on fast storage. Cold can be cheaper. Consider SmartStore when storage growth outpaces compute need.
• In a cluster, keep indexes.conf identical across peers by pushing it from the cluster manager.

Availability

• Cluster only when you have a real availability requirement, not by default.
• Keep an odd number of search head cluster members so captain election always has a majority.
• Keep management roles off the data nodes so they stay responsive during a failure.
• Use different secret keys for the indexer cluster and the search head cluster.

Operations

• Register every node with the Monitoring Console and watch trends, not just moments.
• Size for peak load, not the daily average.
• Test upgrades and configuration changes somewhere safe before production.
• Document your topology, your index design, and your retention rules so the next person can follow them.

Architect checklist

Use this as a fast pass over a design, whether you are building new or reviewing something you inherited. If you cannot answer a line clearly, that is where to dig.

Requirements

• What is the daily ingest volume, at peak, not just average?
• How long must each type of data be retained, and what happens after?
• How many people search, and how many scheduled searches run?
• What downtime and data loss can the business actually accept?

Topology

• Does the deployment shape, single, distributed, or clustered, match the availability requirement?
• Is there a validated architecture this maps to?
• Is it single site or multi-site, and does that match the site-failure risk?

Data in

• Are forwarders universal by default, with heavy forwarders only where justified?
• Is there a deployment server plan for managing the forwarder fleet?
• Are sourcetypes and parsing settings defined explicitly and placed on the right tier?

Storage

• Is index design set by retention, sensitivity, and audience?
• Are replication factor and search factor chosen and understood?
• Is the freeze and archive behavior set so nothing is lost by accident?
• Is SmartStore considered where storage growth outpaces compute?

Sizing

• Is indexer count derived from peak ingest plus search load plus headroom?
• Does storage meet the IOPS the workload needs?
• Is search head count set by user and scheduled search load, with an odd cluster size?

Security and operations

• Is authentication tied to central identity, with least-privilege roles?
• Is traffic encrypted and are secret keys protected and separated?
• Is every node registered with the Monitoring Console?
• Is the whole design documented for the next person?

9.x versus 10.x notes

This handbook targets Splunk Enterprise 9.x, with 9.4 as the reference. Splunk has since shipped the 10.x line, and the current release as of mid 2026 is in the 10.x series, with 9.4.x continuing as a maintained line. The core architecture in this book, the pipeline, the components, clustering, buckets, and the configuration system, is the same across both lines. That is why the concepts here carry forward.

What tends to stay the same

• The four-stage data pipeline and the index-time versus search-time split.
• The component roles: indexers, search heads, forwarders, cluster manager, deployer, deployment server, license manager, Monitoring Console.
• Replication factor and search factor as the core of indexer cluster resilience.
• The captain and deployer model for search head clusters.
• The bucket lifecycle and the configuration file and precedence system.

What tends to move between releases

• Exact reference hardware figures for CPU, memory, IOPS, and storage.
• Supported object storage options and specifics for SmartStore.
• Default values inside conf files, and newly added settings.
• Platform and operating system support, and end-of-life dates for older lines.

Glossary