Shell Scripting

Shell scripting is the art of automating tasks through the Unix shell - combining built-in commands, control flow, and process management to build reliable CLI tools and automation workflows. This skill covers production-quality bash and zsh scripting: robust error handling, portable argument parsing, safe file operations, and the idioms that separate fragile one-liners from scripts that hold up in production.

---

When to use this skill

Trigger this skill when the user:

• Asks to write or review a bash or zsh script
• Needs to parse command-line arguments or flags
• Wants to automate a CLI workflow or task runner
• Asks about exit codes, signal trapping, or error handling in shell
• Needs to process files, lines, or streams from the terminal
• Asks about here documents, process substitution, or subshells
• Wants a portable script that works across bash, zsh, and sh

Do NOT trigger this skill for:

• Python or Node.js CLI tools (shell is the wrong tool for complex logic)
• Scripts that require structured data parsing at scale (use a real language instead)

---

Key principles

1. Always use set -euo pipefail - Start every non-trivial script with this. -e exits on error, -u treats unset variables as errors, -o pipefail catches failures in pipelines. Without this, silent failures hide bugs for weeks.

2. Quote everything - Always double-quote variable expansions: "$var", "$@", "${array[@]}". Unquoted variables break on whitespace and glob characters. The only exceptions are intentional word splitting and arithmetic contexts.

3. Check dependencies upfront - Verify required commands exist before the script runs. Fail fast at the top with a clear error, not halfway through a destructive operation.

4. Use functions for reuse and readability - Extract logic into named functions. Shell functions support local variables (local), can return exit codes, and make scripts testable. A main() function at the bottom with a guard is idiomatic.

5. Prefer shell built-ins over external commands - [[ ]] over [ ], ${var##*/} over basename, ${#str} over wc -c. Built-ins are faster, more portable, and avoid spawning subshells. Use printf over echo for reliable output formatting.

---

Core concepts

Exit codes - Every command returns an integer 0-255. 0 means success; any non-zero value means failure. Use $? to read the last exit code. Use explicit exit N to return meaningful codes from scripts. The || and && operators branch on exit code.

File descriptors - 0 = stdin, 1 = stdout, 2 = stderr. Redirect stderr with 2>file or merge it into stdout with 2>&1. Use >&2 to write errors to stderr so they don't pollute captured output.

Subshells - Parentheses (cmd) run commands in a child process. Changes to variables, cd, or set inside a subshell do not affect the parent. Command substitution $(cmd) also runs in a subshell and captures its stdout.

Variable scoping - All variables are global by default. Use local inside functions to limit scope. declare -r creates read-only variables. declare -a declares arrays; declare -A declares associative arrays (bash 4+).

IFS (Internal Field Separator) - Controls how bash splits words and lines. Default is space/tab/newline. When reading files line by line, set IFS= to prevent trimming of leading/trailing whitespace: while IFS= read -r line.

---

Common tasks

Robust script template with trap cleanup

Every production script should start with this foundation:

#!/usr/bin/env bash
set -euo pipefail

# --- constants ---
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly TMP_DIR="$(mktemp -d)"

# --- cleanup ---
cleanup() {
  local exit_code=$?
  rm -rf "$TMP_DIR"
  if [[ $exit_code -ne 0 ]]; then
    echo "ERROR: $SCRIPT_NAME failed with exit code $exit_code" >&2
  fi
  exit "$exit_code"
}
trap cleanup EXIT INT TERM

# --- dependency check ---
require_cmd() {
  if ! command -v "$1" &>/dev/null; then
    echo "ERROR: required command '$1' not found" >&2
    exit 1
  fi
}
require_cmd curl
require_cmd jq

# --- main logic ---
main() {
  echo "Running $SCRIPT_NAME from $SCRIPT_DIR"
  # ... your logic here
}

main "$@"

The trap cleanup EXIT fires on any exit - success, error, or signal - ensuring temp files are always removed. BASH_SOURCE[0] resolves the script's real location even when called via symlink.

Argument parsing with getopts and long opts

Use getopts for POSIX-portable short flags. For long options, use a while/case loop with manual shift:

usage() {
  cat >&2 <<EOF
Usage: $SCRIPT_NAME [OPTIONS] <input>

Options:
  -o, --output <dir>   Output directory (default: ./out)
  -v, --verbose        Enable verbose logging
  -h, --help           Show this help
EOF
  exit "${1:-0}"
}

OUTPUT_DIR="./out"
VERBOSE=false

parse_args() {
  while [[ $# -gt 0 ]]; do
    case "$1" in
      -o|--output)
        [[ -n "${2-}" ]] || { echo "ERROR: --output requires a value" >&2; usage 1; }
        OUTPUT_DIR="$2"; shift 2 ;;
      -v|--verbose)
        VERBOSE=true; shift ;;
      -h|--help)
        usage 0 ;;
      --)
        shift; break ;;
      -*)
        echo "ERROR: unknown option '$1'" >&2; usage 1 ;;
      *)
        break ;;
    esac
  done
  # remaining positional args available as "$@"
  INPUT_FILE="${1-}"
  [[ -n "$INPUT_FILE" ]] || { echo "ERROR: input file required" >&2; usage 1; }
}

parse_args "$@"

File processing - read, write, and temp files safely

# Read a file line by line without trimming whitespace or interpreting backslashes
while IFS= read -r line; do
  echo "Processing: $line"
done < "$input_file"

# Read into an array
mapfile -t lines < "$input_file"   # bash 4+; equivalent: readarray -t lines

# Write to a file atomically (avoids partial writes on failure)
write_atomic() {
  local target="$1"
  local tmp
  tmp="$(mktemp "${target}.XXXXXX")"
  # write to tmp, then atomically rename
  cat > "$tmp"
  mv "$tmp" "$target"
}
echo "final content" | write_atomic "/etc/myapp/config"

# Safe temp file with auto-cleanup (cleanup trap handles TMP_DIR removal)
local tmpfile
tmpfile="$(mktemp "$TMP_DIR/work.XXXXXX")"
some_command > "$tmpfile"
process_result "$tmpfile"

String manipulation without external tools

# Substring extraction: ${var:offset:length}
str="hello world"
echo "${str:6:5}"        # "world"

# Pattern removal (greedy ##, non-greedy #; greedy %%, non-greedy %)
path="/usr/local/bin/myapp"
echo "${path##*/}"       # "myapp"     (strip longest prefix up to /)
echo "${path%/*}"        # "/usr/local/bin" (strip shortest suffix from /)

# Search and replace
filename="report-2024.csv"
echo "${filename/csv/tsv}"   # "report-2024.tsv"   (first match)
echo "${filename//a/A}"      # "report-2024.csv" -> "report-2024.csv" (all matches)

# Case conversion (bash 4+)
lower="${str,,}"         # all lowercase
upper="${str^^}"         # all uppercase
title="${str^}"          # capitalise first character

# String length and emptiness checks
[[ -z "$var" ]] && echo "empty"
[[ -n "$var" ]] && echo "non-empty"
echo "length: ${#str}"

# Check if string starts/ends with a pattern (no grep needed)
[[ "$str" == hello* ]] && echo "starts with hello"
[[ "$str" == *world ]] && echo "ends with world"

Parallel execution with xargs and GNU parallel

# xargs: run up to 4 jobs in parallel, one arg per job
find . -name "*.log" -print0 \
  | xargs -0 -P4 -I{} gzip "{}"

# xargs with a shell function (must export it first)
process_file() {
  local f="$1"
  echo "Processing $f"
  # ... work ...
}
export -f process_file
find . -name "*.csv" -print0 \
  | xargs -0 -P"$(nproc)" -I{} bash -c 'process_file "$@"' _ {}

# GNU parallel (more features: progress, retry, result collection)
# parallel --jobs 4 --bar gzip ::: *.log
# parallel -j4 --results /tmp/out/ ./process.sh ::: file1 file2 file3

# Manual background jobs with wait
pids=()
for host in "${hosts[@]}"; do
  ssh "$host" uptime &
  pids+=($!)
done
for pid in "${pids[@]}"; do
  wait "$pid" || echo "WARN: job $pid failed" >&2
done

Portable scripts across bash, zsh, and sh

# Detect the running shell
detect_shell() {
  if [ -n "${BASH_VERSION-}" ]; then
    echo "bash $BASH_VERSION"
  elif [ -n "${ZSH_VERSION-}" ]; then
    echo "zsh $ZSH_VERSION"
  else
    echo "sh (POSIX)"
  fi
}

# POSIX-safe array alternative (use positional parameters)
set -- alpha beta gamma
for item do          # equivalent to: for item in "$@"
  echo "$item"
done

# Use $(...) not backticks - both portable, but $() is nestable
result=$(echo "$(date) - $(whoami)")

# Avoid bashisms when targeting /bin/sh:
#   [[ ]] -> [ ]          (but be careful with quoting)
#   local -> still works in most sh implementations (not POSIX but widely supported)
#   readonly var=val      (POSIX-safe)
#   printf not echo -e    (echo -e is not portable)

printf '%s\n' "Safe output with no echo flag issues"

Interactive prompts and colored output

# Color constants (no-op when not a terminal)
setup_colors() {
  if [[ -t 1 ]]; then
    RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
    BLUE='\033[0;34m'; BOLD='\033[1m'; RESET='\033[0m'
  else
    RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; RESET=''
  fi
}
setup_colors

log_info()    { printf "${GREEN}[INFO]${RESET}  %s\n" "$*"; }
log_warn()    { printf "${YELLOW}[WARN]${RESET}  %s\n" "$*" >&2; }
log_error()   { printf "${RED}[ERROR]${RESET} %s\n" "$*" >&2; }

# Yes/no prompt
confirm() {
  local prompt="${1:-Continue?} [y/N] "
  local reply
  read -r -p "$prompt" reply
  [[ "${reply,,}" == y || "${reply,,}" == yes ]]
}

# Prompt with default value
prompt_with_default() {
  local prompt="$1" default="$2" value
  read -r -p "$prompt [$default]: " value
  echo "${value:-$default}"
}

# Spinner for long operations
spin() {
  local pid=$1 msg="${2:-Working...}"
  local frames=('|' '/' '-' '\')
  local i=0
  while kill -0 "$pid" 2>/dev/null; do
    printf "\r%s %s" "${frames[i++ % 4]}" "$msg"
    sleep 0.1
  done
  printf "\r\033[K"  # clear the spinner line
}

---

Gotchas

1. set -e swallows non-zero exits in conditionals - set -e does NOT exit on non-zero returns inside if, while, until, or ||/&& chains. A command like if some_command; then will not trigger -e if some_command fails - this is correct behavior but surprises people who expect -e to be a global safety net.

2. local does not isolate errors from set -e - local var=$(command_that_fails) always returns exit code 0 because local itself succeeds. The subcommand failure is silently swallowed. Declare local var on one line, then var=$(command_that_fails) on the next so set -e can catch it.

3. mktemp without -d creates a file, not a directory - TMP=$(mktemp) creates a temp file. If you then try mkdir "$TMP/subdir" it fails. Use mktemp -d when you need a temp directory.

4. Trap fires on subshell exits too - A trap cleanup EXIT in a parent script also fires when any subshell ( ... ) in that script exits. If your cleanup function deletes temp directories, a subshell exit mid-script can remove files the parent still needs. Use trap selectively or test $BASH_SUBSHELL inside the trap function.

5. Word splitting on array expansion without [@] - "${arr[*]}" expands the array as a single word joined by IFS; "${arr[@]}" expands each element as a separate word. Using * instead of @ when passing arrays to functions causes multi-word elements to silently merge.

---

Anti-patterns

Anti-pattern	Why it's wrong	What to do instead
Missing set -euo pipefail	Errors in pipelines and unset variables are silently ignored, causing downstream data corruption	Add set -euo pipefail as the second line of every script
Unquoted variable: rm -rf $dir	If $dir is empty or contains spaces, the command destroys unintended paths	Always quote: rm -rf "$dir"
Parsing ls output	ls output is designed for humans; filenames with spaces or newlines break word splitting	Use find ... -print0 | xargs -0 or a for f in ./* glob
Using cat file | grep (useless cat)	Spawns an extra process for no reason	Use input redirection: grep pattern file
if [ $? -eq 0 ]	Testing $? after the fact is fragile - any intervening command resets it	Test the command directly: if some_command; then ...
Heredoc with leading whitespace	Indented heredoc content with <<EOF includes the indentation literally	Use <<-EOF to strip leading tabs (not spaces), or use printf

---

References

For detailed reference content, see:

• references/bash-cheatsheet.md - Quick reference for bash built-ins, parameter expansion, test operators, and special variables