Community:SplunkBackupScript Linux

From Splunk Wiki

Backing up Splunk in RHEL Linux

Objective

This script will backup most of the specific splunk customizations. This script will be run as a cron job and create a nightly full backup of the Splunk settings. We also included a flag to do a quick one time backup useful before or after you make major changes and want to fall back to a known working model.

Compatability

Splunk version: 3.4.10

Current Setup

1. Red Hat Enterprise Linux 4
2. Default Splunk 3.4.10 RPM install to

   /opt/splunk

File Locations Used Splunk settings. Most user created settings are in ../etc/system/local/

/opt/splunk/etc/...

Store our backup logs in a new file under this directory.

/var/log/backup/

Location of the actual backups themselves

/var/backup

Script Narrative

The script can be run using the standard linux execution

 ./splunk_backup.sh

. This is the equivalent of running a "quick" backup that is executed before an administrator performs any actions. If you run the standard execution string with

--cron

appended to the end the script will assume you are running the script nightly as part of a cron job. The only difference is if the script will append a time stamp into the filename string. I chose to omit the string to help indicate if the system performed and automated cron backup versus a quick backup.

Once the script is executed with or without the flag it performs the following

1. Determine if using --cron and append a timestamp if you are not using cron
2. Generate a tar file with the appropriate day, time of day.
   1. The script will only backup directories or files specified in the BACKUPLIST variable.
   2. The script will ignore directories or files specified in the IGNORELIST variable.
3. After the tar file is made it echos the contents into the splunk log file
4. If compression is enabled the tar file is gzip'd to a smaller file size
5. Regardless of compression we md5sum the file to ensure if you look at it later it has not been altered. The sum is stored in the splunk log file.
6. If you want the script can remove backups over X number of day old to prevent your disk from filling up.

Supporting Files

backupfiles.txt

This can be any file you choose as long as you update the location in the variables section in the beginning of the script.
Mine looks like this:

/opt/security/splunk_backup.sh
/opt/splunk/etc
/opt/splunk/share/splunk/search_oxiclean/dynamic/html/login.html
/var/backup/backupfiles.txt
/var/backup/ignorefiles.txt

As you can see I want to gather all the files under /opt/splunk/etc as well as the script and supporting files.

ignorefiles.txt

This can be any file you choose as long as you update the location in the variables section in the beginning of the script.
Mine looks like this:

/opt/splunk/etc/system/local/README
/opt/splunk/etc/system/README/*
/opt/splunk/etc/ngram-models/*

As you can see I want to forget backing up the README's and a few other things.

Script

#!/bin/sh
#Backup of the "local" files for Splunk application.
#These are the "brains" of the splunk application

# Setup commmon script options
# These can be modified to fit your needs
# Where should we put the backed up files?
# Default is /var/backup
BACKUPDIR=/var/backup
# Should this script compress the backups?
# Default is ON (YES!)
#
COMPRESS=ON
# Should this script delete old backups?
# Default is OFF (NO!)
#
ROLL=ON
# If the script is going to delete old backups
# How old should the backups be before we delete them?
# Default is 60 Days
#
ROLLAGE=366
# Where should we output the logs of activity?
# Default is /var/log/backup/splunk.log
#
LOGFILE=/var/log/backup/splunk.log
# Where can we find the backupfile list?
# Default is /var/backup/backupfiles.txt
BACKUPLIST=/var/backup/backupfiles.txt
# Where can we find the ignored files list?
# Default is /var/backup/ignorefiles.txt
IGNORELIST=/var/backup/ignorefiles.txt
# Setup our naming convetions based upon date & time formats
#
#
# The dates will look like this: YYYYMMDD
# Example: 20090101 | January 1st 2009
#
PDATE=`date -u +%Y%m%d`
# The times will look like this: HH:MM
# Example: 13:51 | 1PM 51 Minutes
#
PTIME=`date -u +%H:%M`


# Setup script runtime options
if [ "$1" = "--cron" ]
        then
                CRON=1
elif [ "$1" != "" ]
        then
                echo "Invalid script option!"
                                echo "Only valid script option(s) are:"
                                echo "--cron | sets the script to assume once a day cron run"
                                echo "No script options will assume script needs to backup recent changes"
                                echo "This will append a timestamp to the end of the generated backup"
                exit 1
        else
                CRON=0
fi

# Setup script file names
# Ok if this is cron job lets remove the time stamp.
# Ok if this is a normal execution by a human lets add a time stamp.

if [ "$CRON" = "1" ]
        then
                BACKUPFILE=$BACKUPDIR/splunk_"$PDATE".tar
        else
                BACKUPFILE=$BACKUPDIR/splunk_"$PDATE"-"$PTIME".tar
fi

# Lets make a tar-ball

tar -cvf $BACKUPFILE -T $BACKUPLIST -X $IGNORELIST

# List date and of backup into backup log
echo >> $LOGFILE
echo Archive date: $PDATE $PTIME >> $LOGFILE

# List files in tar-ball into backup log
tar -tvf $BACKUPFILE >> $LOGFILE

# Should we gzip the file to reduce space?
# If yes then lets do IT!
# Also lets md5sum the file

if [ "$COMPRESS" = "ON" ]
        then
                gzip $BACKUPFILE
                md5sum $BACKUPFILE.gz >> $LOGFILE
        else
                md5sum $BACKUPFILE >> $LOGFILE
fi

# Look for old files to remove
if [ $ROLL = ON ]
        then
                echo "Deleting these files" >> $LOGFILE
                find $BACKUPDIR/  -mtime +$ROLLAGE >> $LOGFILE
                find $BACKUPDIR/  -mtime +$ROLLAGE -exec rm -f {} \;
                                du -h $BACKUPDIR/ >> $LOGFILE
        else
                echo "No backup files removed" >> $LOGFILE
                echo "Current Sizes" >> $LOGFILE
                du -h $BACKUPDIR/ >> $LOGFILE
fi

if [ $? != 0 ]
        then
                echo "ERRORS!"
                exit 1
        else
                exit 0
fi

Jasonnadeau 08:06, 13 August 2009 (PDT)