Splunk Architect Exercises

The value in local wins, because for the same app, local overrides default. Prove it by running:

./splunk btool props list <sourcetype> --debug

The --debug flag prints the source file next to each winning setting, so you can see the local file is the one being used.

They edited the default folder, which the upgrade replaced. The fix is to put the change in etc/apps/search/local/limits.conf instead. Local folders are yours and survive upgrades; default folders belong to the vendor.

# inputs.conf on the Windows universal forwarder
[WinEventLog://Security]
disabled = 0
index = win_security

[WinEventLog://System]
disabled = 0
index = win_os

Make sure both indexes exist on the indexers, and that the forwarder has an outputs.conf pointing at them.

This is index-time filtering, so it belongs on the indexers or a heavy forwarder, not a universal forwarder.

# transforms.conf
[drop_healthcheck]
REGEX = healthcheck
DEST_KEY = queue
FORMAT = nullQueue

# props.conf
[web_access]
TRANSFORMS-drop = drop_healthcheck

Route everything to null first, then route the keepers back to the index queue. The last matching transform wins, so order matters.

# transforms.conf
[set_null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keep_errors]
REGEX = (ERROR|FATAL)
DEST_KEY = queue
FORMAT = indexQueue

# props.conf
[app_log]
TRANSFORMS-filter = set_null, keep_errors

Index-time masking runs on the indexers or a heavy forwarder. Use SEDCMD:

# props.conf
[payments]
SEDCMD-mask = s/\d{12}(\d{4})/xxxxxxxxxxxx\1/g

Set this before the source goes live. Masking cannot clean data that is already indexed.

Data survives the loss of replication factor minus 1 peers, which is 2. Full searchability survives the loss of search factor minus 1 peers, which is 1, after which the cluster rebuilds searchable copies from the remaining raw copies. So it can lose 2 peers without losing data, and 1 peer without any gap in searchable copies.

Three members. The captain is chosen by majority vote, so you need an odd number that can still form a majority after a loss. With two members, losing one leaves a single member that cannot hold a majority, so the cluster cannot safely elect a captain. Three survives one loss and keeps a majority of two.

1. Set the deployer secret and label in server.conf on the deployer, then restart it.
2. Run init shcluster-config on each member, then restart each.
3. Run bootstrap shcluster-captain on one member, listing all members.
4. Push apps with apply shcluster-bundle from the deployer.

The order matters because members must be initialised before the cluster can be bootstrapped, and the deployer must be ready before you push apps.

Configure a static captain so the survivors can work. On one healthy member:

./splunk edit shcluster-config -election false -mode captain \
    -captain_uri https://sh1:8089 -auth admin:pass

On the other reachable member:

./splunk edit shcluster-config -election false -mode member \
    -captain_uri https://sh1:8089 -auth admin:pass

When enough members return to form a majority, turn election back on with -election true on every member and restart, so the cluster manages the captain dynamically again.

The risk: joining an instance to the cluster overwrites its existing apps and config with the cluster's. Do not add a search head that holds content you need unless that content is already in the deployer.

Steps: initialise the new instance with init shcluster-config using the same settings as the others and restart it, then join it. From the new member:

./splunk add shcluster-member -current_member_uri https://sh1:8089 -auth admin:pass

Going from three to four makes the count even, so plan to reach five, or be aware that an even count is less ideal for election.

640 divided by 100 is 6.4, so you need 7 indexers to cover peak ingest, since you round up. This is ingest capacity only. You would then add indexers for search concurrency and for headroom, and account for replication overhead if you cluster. The 100 GB figure here is an assumption for the exercise, not a fixed Splunk value; always use the real per-indexer capacity for your spec and workload.

Estimate stored size as daily volume times retention days, adjusted for compression and index overhead. Raw data usually compresses well, but the index files add some of that back, so architects often plan using a total on-disk figure per day rather than the raw log size. For 30 days at 200 GB per day of raw data, start from 6 TB of raw and apply your measured on-disk ratio for that data. Clustering then multiplies this by the replication factor, because each bucket is stored that many times, plus extra for searchable copies. So a replication factor of 3 roughly triples the disk. Measure your own compression ratio rather than assuming one.

SmartStore. It keeps the master copy of data in cheap remote object storage and uses local disk as a cache for the working set. This decouples storage from compute, so the team grows object storage for capacity and adds indexers only when they need more ingest or search power. Sizing shifts from sizing local disk for all retained data to sizing the cache for the data searches actually touch, plus sizing the object store for total retention.

Use a clustered distributed design:

An indexer cluster with enough peers for peak ingest plus search plus headroom, replication factor at least 3 and search factor 2 so it survives a peer loss. A search head cluster of at least three members for the analysts and availability. A forwarder tier of universal forwarders feeding the peers, managed by a deployment server.

Management nodes, each kept off the data nodes: a cluster manager for the indexer cluster, a deployer for the search head cluster, a central license manager, and a Monitoring Console. Consider SmartStore if storage growth will outpace compute.

1. On the forwarder, confirm the input exists: ./splunk list monitor.
2. On the forwarder, confirm it points at the indexers: ./splunk list forward-server.
3. On the indexer, confirm the receiving port is enabled: check ./splunk display listen or that [splunktcp://9997] is set.
4. Confirm the target index exists on the indexers.
5. Read splunkd.log on both sides, or search index=_internal, for connection errors such as a blocked port.
6. Confirm the network path and firewall allow the forwarder to reach 9997.

./splunk btool <conf> list <stanza> --debug

The --debug flag is the key part. It prints the exact file each winning setting came from, so you can spot a higher-precedence copy overriding yours, or find that your setting is on the wrong tier.

Do not restart all peers at once. Push the change as a bundle from the cluster manager, and use the rolling restart action so peers restart one at a time while the others keep the data available. Let the cluster return to a healthy state between restarts. Check status with ./splunk show cluster-status on the manager before and after.