Upgrade procedure:

Splunk Upgrade Guide

This document provides a step-by-step procedure to upgrade Splunk from an existing version to a newer release. Follow each section carefully to ensure a smooth upgrade process.

Table of Contents

  1. Preparation
  2. Download and Transfer Files
  3. Pre-Upgrade Checks
  4. Device State Management
  5. Backup Current Configuration
  6. Install the New Version
  7. Post-Upgrade Verification
  8. Documentation and Communication

1. Preparation

  • Ensure you have access to the target devices and necessary permissions.
  • Confirm the current Splunk version.
  • Notify customer and colleagues about the upcoming upgrade.
  • Review release notes for the target version.

2. Download and Transfer Files

Download the Splunk Installer

You can use the wget command to download the package on the host wget -O splunk---Linux-x86_64.tgz “https://download.splunk.com/products/splunk/releases//linux/splunk---Linux-x86_64.tgz”

sudo -i
cd /tmp
wget -O splunk-9.4.3-237ebbd22314-linux-amd64.tgz "https://download.splunk.com/products/splunk/releases/9.4.3/linux/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

Transfer the Installer to Target Devices

Use WinSCP or similar tools to connect and send to the device: put “path\to\splunk---Linux-x86_64.tgz” “/tmp/splunk---Linux-x86_64.tgz”

set path=C:\Program Files (x86)\WinSCP;%path%
winscp sftp://<CLI-account>@<device_NAT_IP>:/tmp
put "path\to\splunk-9.4.3-237ebbd22314-linux-amd64.tgz" "/tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

Use SCP on more recent windows machines or on Linux:

scp "path\to\splunk-9.4.3-237ebbd22314-linux-amd64.tgz" user@remote-ip:"/tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

3. Pre-Upgrade Checks

Check if Splunk has an active process:

ps -ef | grep splunk

Check if Splunk is running:

/opt/splunk/bin/splunk status

Check Splunk version:

/opt/splunk/bin/splunk --version

4. Device State Management

Check whether a monitoring needs to set the host in maintenance mode

On Splunk in environments with index clusters: Maintenance mode halts most bucket fixup activity and prevents frequent rolling of hot buckets. It is useful when performing peer upgrades and other maintenance activities on an indexer cluster. Enable maintenance mode:

/opt/splunk/bin/splunk enable maintenance-mode

Disable maintenance mode:

/opt/splunk/bin/splunk disable maintenance-mode

Enable maintenance mode:

/opt/splunk/bin/splunk show maintenance-mode

5. Backup Current Configuration

Check KV Store status

/opt/splunk/bin/splunk show kvstore-status

Backup KV Store

/opt/splunk/bin/splunk backup kvstore

Check backup

ls -la /opt/splunk/var/lib/splunk/kvstorebackup/

Backup config files

sudo -i
cd /tmp
mkdir /tmp/etc_backup$(date +"%d-%m-%Y")
cp -a /opt/splunk/etc/ /tmp/etc_backup$(date +"%d-%m-%Y")/

Check backup

ls /tmp/etc_backup$(date +"%d-%m-%Y")/etc

6. Install the New Version

stop splunk using Splunk command

/opt/splunk/bin/splunk stop

stop splunk using systemctl command

systemctl stop splunk

Unpack the Installer

tar -xvzf /tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz -C /opt/

Change the owner of the Splunk folder recursively with the account running Splunk

chown -R splunk.splunk /opt/splunk

Since I’m using a splunk user account (created without elevated privileges) to run splunk, I am switching to the account

su - splunk

Start Splunk with License Acceptance

/opt/splunk/bin/splunk start --accept-license --answer-yes

7. Post-Upgrade Verification

Check if Splunk is running

/opt/splunk/bin/splunk status

Check recent logs for errors

tail -20f /opt/splunk/var/log/splunk/splunkd.log

Check Splunk version

/opt/splunk/bin/splunk --version

Check if Splunk service is run by splunk user

ps -ef | grep splunkd | grep -v grep

8. Documentation and Communication

  • Update internal documentation with the new version details.
  • Notify customer and colleagues of the successful upgrade.
  • Share release notes and any post-upgrade instructions.