Errorcheck within Splunk GUI

index=_internal log_level=ERROR source="/opt/splunk/var/log/splunk/splunkd.log"

Check Splunk version

| rest splunk_server=* count=1 /services/server/info 
| table version host

List indexes with the retention period

| rest splunk_server=* /services/data/indexes 
| eval "Retention Period (months)"=round((frozenTimePeriodInSecs/2628000),0)
| search NOT title IN ("_*", "main", "history", "summary", "splunklogger") 
| table title "Retention Period (months)" 
| rename title as Index
```for years, divide frozenTimePeriodInSecs/31556952; for days, divide frozenTimePeriodInSecs/86400```

Overview of hosts sending logs by index

| tstats values(host) where index=* by index

Usage per day by index + percentages

index=_internal source=*license_usage.log type="Usage"
| bin _time span=1d
| stats sum(b) as b by idx, _time
| eval GB=round((b/31556952),2)
| eventstats sum(GB) as total_GB by _time
| eval percentage=round((GB/total_GB)*100,2)
| sort -_time -GB
| fields - b, total_GB
| rename idx as Index, GB as "Size (GB)", percentage as "Percentage (%)"
| convert timeformat="%Y-%m-%d" ctime(_time) as Date
| fields Date, Index, "Size (GB)", "Percentage (%)"
```for years, divide b/31556952; for days, divide b/31556952```

List users and their roles

| rest /services/authentication/users splunk_server=local
| fields title roles email
| rename title as username

Start Splunk

/opt/splunk/bin/splunk start

Stop Splunk

/opt/splunk/bin/splunk stop

Restart Splunk

/opt/splunk/bin/splunk restart

Check if Splunk is running

/opt/splunk/bin/splunk status

Reload a serverclass to push deployment apps without restarting Splunk"

/opt/splunk/bin/splunk reload deploy-server -class [serverclass-name]

Check license expiration date

/opt/splunk/bin/splunk list licenses | grep "expiration_time" | awk -F':' '{print $2}' | xargs -I{} date -d @{} +"%Y-%m-%d %H:%M:%S"

List installed apps and their status

/opt/splunk/bin/splunk list app

List installed apps and their version (if found)

/opt/splunk/bin/splunk list app | grep version /opt/splunk/etc/apps/*/default/app.conf

Install an app

/splunk install app <path to app.package>

Update an app

/splunk install app <path to app.package> -update 1

Remove an app

/opt/splunk/bin/splunk remove app [appname]

Check Splunk admins

/opt/splunk/bin/splunk list user | grep admin -B2

Enable Splunk service to start when the host boots up

/opt/splunk/bin/splunk enable

Disable Splunk service so it doesn’t start when the host boots up

/opt/splunk/bin/splunk disable

Find the startup message

cat /opt/splunk/var/log/splunk/splunkd_stdout.log | grep "Splunk>" | tail -n 1

Splunk Upgrade Guide

This document provides a step-by-step procedure to upgrade Splunk from an existing version to a newer release. Follow each section carefully to ensure a smooth upgrade process.

Table of Contents

1. Preparation
2. Download and Transfer Files
3. Pre-Upgrade Checks
4. Device State Management
5. Backup Current Configuration
6. Install the New Version
7. Post-Upgrade Verification
8. Documentation and Communication

1. Preparation

• Ensure you have access to the target devices and necessary permissions.
• Confirm the current Splunk version.
• Notify customer and colleagues about the upcoming upgrade.
• Review release notes for the target version.

2. Download and Transfer Files

Download the Splunk Installer

You can use the wget command to download the package on the host wget -O splunk---Linux-x86_64.tgz “https://download.splunk.com/products/splunk/releases//linux/splunk---Linux-x86_64.tgz”

sudo -i
cd /tmp
wget -O splunk-9.4.3-237ebbd22314-linux-amd64.tgz "https://download.splunk.com/products/splunk/releases/9.4.3/linux/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

Transfer the Installer to Target Devices

Use WinSCP or similar tools to connect and send to the device: put “path\to\splunk---Linux-x86_64.tgz” “/tmp/splunk---Linux-x86_64.tgz”

set path=C:\Program Files (x86)\WinSCP;%path%
winscp sftp://<CLI-account>@<device_NAT_IP>:/tmp
put "path\to\splunk-9.4.3-237ebbd22314-linux-amd64.tgz" "/tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

Use SCP on more recent windows machines or on Linux:

scp "path\to\splunk-9.4.3-237ebbd22314-linux-amd64.tgz" user@remote-ip:"/tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz"

3. Pre-Upgrade Checks

Check if Splunk has an active process:

ps -ef | grep splunk

Check if Splunk is running:

/opt/splunk/bin/splunk status

Check Splunk version:

/opt/splunk/bin/splunk --version

4. Device State Management

Check whether a monitoring needs to set the host in maintenance mode

On Splunk in environments with index clusters: Maintenance mode halts most bucket fixup activity and prevents frequent rolling of hot buckets. It is useful when performing peer upgrades and other maintenance activities on an indexer cluster. Enable maintenance mode:

/opt/splunk/bin/splunk enable maintenance-mode

Disable maintenance mode:

/opt/splunk/bin/splunk disable maintenance-mode

Enable maintenance mode:

/opt/splunk/bin/splunk show maintenance-mode

5. Backup Current Configuration

Check KV Store status

/opt/splunk/bin/splunk show kvstore-status

Backup KV Store

/opt/splunk/bin/splunk backup kvstore

Check backup

ls -la /opt/splunk/var/lib/splunk/kvstorebackup/

Backup config files

sudo -i
cd /tmp
mkdir /tmp/etc_backup$(date +"%d-%m-%Y")
cp -a /opt/splunk/etc/ /tmp/etc_backup$(date +"%d-%m-%Y")/

Check backup

ls /tmp/etc_backup$(date +"%d-%m-%Y")/etc

6. Install the New Version

stop splunk using Splunk command

/opt/splunk/bin/splunk stop

stop splunk using systemctl command

systemctl stop splunk

Unpack the Installer

tar -xvzf /tmp/splunk-9.4.3-237ebbd22314-linux-amd64.tgz -C /opt/

Change the owner of the Splunk folder recursively with the account running Splunk

chown -R splunk.splunk /opt/splunk

Since I’m using a splunk user account (created without elevated privileges) to run splunk, I am switching to the account

su - splunk

Start Splunk with License Acceptance

/opt/splunk/bin/splunk start --accept-license --answer-yes

7. Post-Upgrade Verification

Check if Splunk is running

/opt/splunk/bin/splunk status

Check recent logs for errors

tail -20f /opt/splunk/var/log/splunk/splunkd.log

Check Splunk version

/opt/splunk/bin/splunk --version

Check if Splunk service is run by splunk user

ps -ef | grep splunkd | grep -v grep

8. Documentation and Communication

• Update internal documentation with the new version details.
• Notify customer and colleagues of the successful upgrade.
• Share release notes and any post-upgrade instructions.

Find a file or directory

find / -type f -name "filename" 2>/dev/null
find / -type d -name "dirname" 2>/dev/null

find files that contain a pattern

find . -type f -exec grep -l 'version' {} \;

Find a file or directory

find / -type f -name "filename" 2>/dev/null
find / -type d -name "dirname" 2>/dev/null

check if machine is a vm or barebone

dmidecode -s system-manufacturer

check folder disk usage

du -hs * | sort -h

check open port

(echo > /dev/tcp/10.254.4.54/22) >/dev/null 2>&1 && echo "It's up" || echo "It's down"

check for listening ports

sudo lsof -i -P -n | grep LISTEN
sudo netstat -tulpn | grep LISTEN
sudo ss -tulpn | grep LISTEN
sudo lsof -i:22 ## see a specific port such as 22 ##
sudo nmap -sTU -O IP-address-Here

Check expiry date on .pem file

openssl x509 -enddate -noout -in /path/to/certificate.pem

dry-run an update on OS

sudo yum check-update

Check if a reboot is required

needs-restarting -r

remove cache of updates for old data

rm -rf /var/cache/yum

block icmp

sysctl -w net.ipv4.icmp_echo_ignore_all=1

remove empty lines from file

sed -i '/^$/d' <filename>

change to root

sudo -i

Grep sudo users

rm -f /tmp/names; for user in $(getent passwd | cut -d: -f1); do count=$((count+1)); if sudo -l -U "$user" | grep -q "ALL"; then echo "$user" >> /tmp/names; echo "Checked $count of $(getent passwd | cut -d: -f1 | wc -l) users."; fi; done; clear; cat /tmp/names; rm -f /tmp/names

This cheat sheet provides an extensive list of Logrotate configuration directives, their descriptions, and examples.
Use this as a quick reference to master log rotation on Unix-like systems.

Each configuration block is tied to a log file or set of log files. Example:

/var/log/example.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0640 root adm
}

Basic Settings:

rotate <count>
Keep number of old log files before deleting them.

daily | weekly | monthly | yearly
Frequency of rotation.

Compression Options:

compress
Compress old versions of log files with gzip.

nocompress
Do not compress old logs.

delaycompress
Postpone compression to the next rotation cycle (used with compress).

File Handling:

missingok
Ignore missing log files and don’t issue an error.

notifempty
Do not rotate the log if it is empty.

ifempty
Rotate the log even if it is empty (default behavior).

create <mode> <owner> <group>
Create a new log file with specified permissions.

copy
Make a copy of the log file and truncate the original.

copytruncate
Truncate the original log file after copying it (useful for active logs).

Date & Naming:

dateext
Append an extension with the current date to rotated log files.

dateformat .%Y-%m-%d
Custom format for dateext (e.g., .2025-06-06).

extension <ext>
Force specific extension for rotated files (e.g., .log).

Size-Based Rotation:

maxage <days>
Remove rotated logs older than .

minsize <size>
Rotate only if log size is above .

size <size>
Rotate if log file size meets threshold, regardless of time.

maxsize <size>
Do not rotate if log is larger than specified size.

Directory & Scripts:

olddir <dir>
Move rotated logs to a specified directory.

sharedscripts
Run postrotate script once for all matching logs.

postrotate/endscript
Script to run after log rotation.

prerotate/endscript
Script to run before log rotation.

firstaction/endscript
Run only once before rotation begins (before prerotate).

lastaction/endscript
Run once after rotation finishes (after postrotate).

tabooext + <ext>
Treat additional extensions as taboo (not rotated).

• Run logrotate -d <config> to debug your config without applying changes.
• Use logrotate -f <config> to force rotation for testing.
• Logrotate is typically triggered via cron or systemd timers.
• Keep your config DRY by centralizing shared logic in /etc/logrotate.conf and using includes.

Create a sudo user that won’t prompt for password on executing sudo commands.

#!/bin/bash

# Ensure script is run as root
if [[ $EUID -ne 0 ]]; then
   echo "❌ This script must be run as root"
   exit 1
fi

# Prompt for new username
read -p "Enter new username: " username

# Check if user already exists
if id "$username" &>/dev/null; then
    echo "⚠️ User '$username' already exists."
    exit 1
fi

# Prompt for password (silent input)
read -s -p "Enter password for $username: " password
echo
read -s -p "Confirm password: " password_confirm
echo

# Check passwords match
if [[ "$password" != "$password_confirm" ]]; then
    echo "❌ Passwords do not match."
    exit 1
fi

# Create user with home directory and bash shell
useradd -m -s /bin/bash "$username"

# Set user password
echo "${username}:${password}" | chpasswd

# Add user to sudo group
usermod -aG sudo "$username"

# Create a sudoers file to allow passwordless sudo
echo "$username ALL=(ALL) NOPASSWD:ALL" > "/etc/sudoers.d/$username"
chmod 440 "/etc/sudoers.d/$username"

echo "✅ User '$username' created with bash shell and passwordless sudo access."

Ncdu is a disk usage analyzer with a text-mode user interface. It is designed to find space hogs on a remote server where you don’t have an entire graphical setup available, but it is a useful tool even on regular desktop systems. Ncdu aims to be fast, simple, easy to use, and should be able to run on any POSIX-like system.
install:

wget https://dev.yorhel.nl/download/ncdu-2.8.1-linux-x86_64.tar.gz
tar -xzf ncdu-2.8.1-linux-x86_64.tar.gz
sudo mv ncdu /usr/local/bin/

usage:

ncdu <directory_name>