Troubleshooting:

Errorcheck within Splunk GUI

index=_internal log_level=ERROR source="/opt/splunk/var/log/splunk/splunkd.log"

Check Splunk version

| rest splunk_server=* count=1 /services/server/info 
| table version host
Index

List indexes with the retention period

| rest splunk_server=* /services/data/indexes 
| eval "Retention Period (months)"=round((frozenTimePeriodInSecs/2628000),0)
| search NOT title IN ("_*", "main", "history", "summary", "splunklogger") 
| table title "Retention Period (months)" 
| rename title as Index
```for years, divide frozenTimePeriodInSecs/31556952; for days, divide frozenTimePeriodInSecs/86400```
Hosts

Overview of hosts sending logs by index

| tstats values(host) where index=* by index
License

Usage per day by index + percentages

index=_internal source=*license_usage.log type="Usage"
| bin _time span=1d
| stats sum(b) as b by idx, _time
| eval GB=round((b/31556952),2)
| eventstats sum(GB) as total_GB by _time
| eval percentage=round((GB/total_GB)*100,2)
| sort -_time -GB
| fields - b, total_GB
| rename idx as Index, GB as "Size (GB)", percentage as "Percentage (%)"
| convert timeformat="%Y-%m-%d" ctime(_time) as Date
| fields Date, Index, "Size (GB)", "Percentage (%)"
```for years, divide b/31556952; for days, divide b/31556952```
Users

List users and their roles

| rest /services/authentication/users splunk_server=local
| fields title roles email
| rename title as username
Maintenance:

Start Splunk

/opt/splunk/bin/splunk start

Stop Splunk

/opt/splunk/bin/splunk stop

Restart Splunk

/opt/splunk/bin/splunk restart

Check if Splunk is running

/opt/splunk/bin/splunk status

Reload a serverclass to push deployment apps without restarting Splunk"

/opt/splunk/bin/splunk reload deploy-server -class [serverclass-name]

Check license expiration date

/opt/splunk/bin/splunk list licenses | grep "expiration_time" | awk -F':' '{print $2}' | xargs -I{} date -d @{} +"%Y-%m-%d %H:%M:%S"

List installed apps and their status

/opt/splunk/bin/splunk list app

List installed apps and their version (if found)

/opt/splunk/bin/splunk list app | grep version /opt/splunk/etc/apps/*/default/app.conf

Install an app

/splunk install app <path to app.package>

Update an app

/splunk install app <path to app.package> -update 1

Remove an app

/opt/splunk/bin/splunk remove app [appname]

Check Splunk admins

/opt/splunk/bin/splunk list user | grep admin -B2
Basic config:

Enable Splunk service to start when the host boots up

/opt/splunk/bin/splunk enable

Disable Splunk service so it doesn’t start when the host boots up

/opt/splunk/bin/splunk disable
Extra:

Find the startup message

cat /opt/splunk/var/log/splunk/splunkd_stdout.log | grep "Splunk>" | tail -n 1
Upgrade procedure:

Splunk Upgrade Guide

This document provides a step-by-step procedure to upgrade Splunk from an existing version to a newer release. Follow each section carefully to ensure a smooth upgrade process.

Table of Contents

  1. Preparation
  2. Download and Transfer Files
  3. Pre-Upgrade Checks
  4. Device State Management
  5. Backup Current Configuration
  6. Install the New Version
  7. Post-Upgrade Verification
  8. Documentation and Communication

1. Preparation

  • Ensure you have access to the target devices and necessary permissions.
  • Confirm the current Splunk version.
  • Notify customer and colleagues about the upcoming upgrade.
  • Review release notes for the target version.

2. Download and Transfer Files

Download the Splunk Installer

You can use the wget command to download the package on the host wget -O splunk---Linux-x86_64.tgz “https://download.splunk.com/products/splunk/releases//linux/splunk---Linux-x86_64.tgz”

sudo -i
cd /tmp
wget -O splunk-9.4.3-237ebbd22314-linux-amd64.tgz "https://download.splunk.com/products/splunk/releases/9.4.3/linux/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

Transfer the Installer to Target Devices

Use WinSCP or similar tools to connect and send to the device: put “path\to\splunk---Linux-x86_64.tgz” “/tmp/splunk---Linux-x86_64.tgz”

set path=C:\Program Files (x86)\WinSCP;%path%
winscp sftp://<CLI-account>@<device_NAT_IP>:/tmp
put "path\to\splunk-9.4.3-237ebbd22314-linux-amd64.tgz" "/tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

Use SCP on more recent windows machines or on Linux:

scp "path\to\splunk-9.4.3-237ebbd22314-linux-amd64.tgz" user@remote-ip:"/tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

3. Pre-Upgrade Checks

Check if Splunk has an active process:

ps -ef | grep splunk

Check if Splunk is running:

/opt/splunk/bin/splunk status

Check Splunk version:

/opt/splunk/bin/splunk --version

4. Device State Management

Check whether a monitoring needs to set the host in maintenance mode

On Splunk in environments with index clusters: Maintenance mode halts most bucket fixup activity and prevents frequent rolling of hot buckets. It is useful when performing peer upgrades and other maintenance activities on an indexer cluster. Enable maintenance mode:

/opt/splunk/bin/splunk enable maintenance-mode

Disable maintenance mode:

/opt/splunk/bin/splunk disable maintenance-mode

Enable maintenance mode:

/opt/splunk/bin/splunk show maintenance-mode

5. Backup Current Configuration

Check KV Store status

/opt/splunk/bin/splunk show kvstore-status

Backup KV Store

/opt/splunk/bin/splunk backup kvstore

Check backup

ls -la /opt/splunk/var/lib/splunk/kvstorebackup/

Backup config files

sudo -i
cd /tmp
mkdir /tmp/etc_backup$(date +"%d-%m-%Y")
cp -a /opt/splunk/etc/ /tmp/etc_backup$(date +"%d-%m-%Y")/

Check backup

ls /tmp/etc_backup$(date +"%d-%m-%Y")/etc

6. Install the New Version

stop splunk using Splunk command

/opt/splunk/bin/splunk stop

stop splunk using systemctl command

systemctl stop splunk

Unpack the Installer

tar -xvzf /tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz -C /opt/

Change the owner of the Splunk folder recursively with the account running Splunk

chown -R splunk.splunk /opt/splunk

Since I’m using a splunk user account (created without elevated privileges) to run splunk, I am switching to the account

su - splunk

Start Splunk with License Acceptance

/opt/splunk/bin/splunk start --accept-license --answer-yes

7. Post-Upgrade Verification

Check if Splunk is running

/opt/splunk/bin/splunk status

Check recent logs for errors

tail -20f /opt/splunk/var/log/splunk/splunkd.log

Check Splunk version

/opt/splunk/bin/splunk --version

Check if Splunk service is run by splunk user

ps -ef | grep splunkd | grep -v grep

8. Documentation and Communication

  • Update internal documentation with the new version details.
  • Notify customer and colleagues of the successful upgrade.
  • Share release notes and any post-upgrade instructions.