Capture errors in shell scripts

Overview

Teaching: 10 min
Exercises: 10 min

Questions

• How the shell helps to keep you safe?

Objectives

• How to prevent common errors when working with Bash scripts

This section describes some Bash options useful when working with scripts in Linux. These are specially important in the context of SLURM job scripts since they help to prevent time consuming errors (e.g. a job waiting in the queue for hours and then crashing in the first minutes or seconds of execution due to variable typo), or more dangerously, undesired data deletion.

The Shell

The Shell is a Linux program for command interpretation, it takes commands from the keyboard and sends them to the operating system for execution. Users interact with the shell when working on a terminal. In the old days of Linux, it was the only_ user interface available. There are several Shells available, the one used in Hawk is Bash (Bourne Again Shell).

Bash shell is a programming language, and as such, it has its own syntax rules and lots of options and features. In this lesson we will only focus on two of them:

• how to trap undefined variables
• how to trap error messages

Trapping undefined variables

Bash has a built-in command set which control shell attributes. In particular, we are interested in set -u. From the manual:

-u      Treat  unset  variables  and  parameters other than the special
        parameters "@" and "*" as an error  when  performing  parameter
        expansion.   If  expansion is attempted on an unset variable or
        parameter, the shell prints  an  error  message,  and,  if  not
        interactive, exits with a non-zero status.

When used within a shell script, set -u will trigger and error if an undefined variable is found.

Define your variables.

• Run lesson_3/trap_1.sh – what do you notice?
• Run lesson_3/trap_2.sh – is this an improvement?

Exit on non-zero exit status

The second useful Bash feature is set -e. From the manual:

-e      Exit immediately if a pipeline (which may consist of  a  single
        simple  command),   a subshell command enclosed in parentheses,
        or one of the commands executed  as  part  of  a  command  list
        enclosed  by braces (see SHELL GRAMMAR above) exits with a non-
        zero status.  The shell does not exit if the command that fails
        is  part  of  the command list immediately following a while or
        until keyword, part of  the  test  following  the  if  or  elif
        reserved words, part of any command executed in a && or || list
        except the command following the final && or ||, any command in
        a  pipeline  but  the last, or if the command's return value is
        being inverted with !.  A trap on  ERR,  if  set,  is  executed
        before the shell exits.  This option applies to the shell envi‐
        ronment and each subshell environment separately  (see  COMMAND
        EXECUTION  ENVIRONMENT  above), and may cause subshells to exit
        before executing all the commands in the subshell.

In short, Bash will stop the program execution if an unhandled error in found. The common way to handle errors in Bash is through conditionals.

Catch the error

• Run lesson_3/trap_3.sh – what is now happening?
• Run lesson_3/trap_4.sh – how is this helping?

Syntax highlighting

A side note on syntax highlighting. It is very useful and some text editors provide it as default (Vim, Emacs). Nano also provide it for a few languages:

To activate it for Bash, create a .nanorc file in your home directory with the following line:

include /usr/share/nano/sh.nanorc

Open any of the previous bash examples, see the difference?

Handling errors

As seen before, undefined variables are easy to fix, but how can errors be fixed? Any error in a program that occurs within an if condition is not trapped by set -e since it is being handled.

if mkdir $MYPATH
then
  echo “INFO: Directory created.”
else
  echo “ERROR: Failed to create directory.”
  exit 1
fi

Any program in a Boolean expression is not trapped since it is also being handled.

mkdir $MYPATH || ( echo “ERROR: Failed to created directory.” && exit 1 ) 

Handle the error

• Run lesson_3/error_1.sh – what do you notice?
• Run lesson_3/error_2.sh – is this an improvement?

Functions

As in any programming language splitting up your tasks makes reading it easier. A shell function is like running a mini version of a script inside another script. For example:

create_directory ()
{
  if mkdir $1
  then 
    echo “INFO: Created directory $1”
  else
    echo “ERROR: Failed to created $1”
  fi
}

Shell functions

• Run lesson_3/function_1.sh – what is wrong with this script?
• Run lesson_3/function_2.sh – how is this helping?

How does this help SLURM

1. Load the modules before any use of trapping and undefined variables.
2. Define functions for specific issues
   • Create working directory
   • Run a particular pre-processing step
   • Process output files
3. Make sure SLURM jobs exits soon after error.
   • If directory is not creatable
   • Input file not found
4. Reduces incorrect runs and reduces jobs that do not produce useful output.

Key Points

• Bash has built-in options that allow to check before executing if variables are set in a script.

• Trapping errors early is very important and can save time and effort in the long run.

• Writing maintainable shell scripts makes it easier to come back and read your code.