Shell Scripting Essentials for Modern Developers
Practical shell scripting patterns for automating development workflows, from basic Bash constructs to robust production scripts.
Shell scripting sits in an uncomfortable middle ground. It is too important to ignore — every deployment pipeline, development setup, and automation workflow touches it. But it is also riddled with footguns that make "simple" scripts silently break in production.
The difference between a script that works on your machine and one that works everywhere comes down to a few foundational practices that most developers skip.
Start Every Script Right
The first three lines of any Bash script determine whether it fails loudly or silently corrupts data.
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'What each line does:
set -e— Exit immediately on any command failure (instead of continuing with bad state)set -u— Treat unset variables as errors (instead of silently expanding to empty string)set -o pipefail— A pipeline fails if any command in it fails (instead of only checking the last)IFS=$'\n\t'— Prevent word splitting on spaces in filenames
# ❌ Without set -euo pipefail — silently uses wrong directory
cd /tmp/deploy
rm -rf *
# If cd fails, this runs rm -rf in whatever directory you're in
# ✅ With set -euo pipefail — stops immediately
set -euo pipefail
cd /tmp/deploy
rm -rf ./*
# If cd fails, the script stops. Crisis averted.That cd failure scenario has caused real production outages. The set -e flag is not optional — it is the seatbelt.
Variables and Quoting
Unquoted variables are the number one source of shell script bugs. A variable containing spaces or globbing characters will expand in ways you did not intend.
# ❌ Unquoted variable — breaks on spaces and special chars
file_path=/tmp/my project/data.csv
cp $file_path /backup/
# Actually runs: cp /tmp/my project/data.csv /backup/
# Shell sees three arguments: /tmp/my, project/data.csv, /backup/
# ✅ Always quote variables
file_path="/tmp/my project/data.csv"
cp "$file_path" /backup/Use "${variable}" for string interpolation and "$@" to pass arguments through to other commands:
# Pass all script arguments to another command
run_tests() {
local test_dir="${1:-.}"
local flags=("${@:2}")
echo "Running tests in ${test_dir}..."
pytest "$test_dir" "${flags[@]}"
}
run_tests "$@"The local keyword scopes variables to the function. Without it, every variable is global — a recipe for accidental overwrites in longer scripts.
Functions and Error Handling
Structure scripts with functions, a main entrypoint, and explicit error handling. This pattern scales from 10-line utilities to 500-line deployment scripts.
#!/usr/bin/env bash
set -euo pipefail
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly LOG_FILE="/tmp/deploy-$(date +%Y%m%d-%H%M%S).log"
log() {
local level="$1"
shift
echo "[$(date +'%H:%M:%S')] [${level}] $*" | tee -a "$LOG_FILE"
}
cleanup() {
local exit_code=$?
if [[ $exit_code -ne 0 ]]; then
log "ERROR" "Script failed with exit code ${exit_code}"
log "ERROR" "Check log: ${LOG_FILE}"
fi
# Remove temp files, restore state, etc.
rm -f /tmp/deploy-lock
}
check_prerequisites() {
local missing=()
for cmd in docker kubectl jq; do
if ! command -v "$cmd" &>/dev/null; then
missing+=("$cmd")
fi
done
if [[ ${#missing[@]} -gt 0 ]]; then
log "ERROR" "Missing required tools: ${missing[*]}"
exit 1
fi
}
main() {
trap cleanup EXIT
log "INFO" "Starting deployment..."
check_prerequisites
# ... rest of deployment logic
log "INFO" "Deployment complete"
}
main "$@"The trap cleanup EXIT ensures cleanup runs whether the script succeeds or fails. The readonly keyword prevents accidental reassignment of constants.
Conditional Logic and Comparisons
Bash has two comparison syntaxes: [ ] (POSIX) and [[ ]] (Bash extended). Use [[ ]] — it handles spaces, pattern matching, and logical operators without surprises.
# ❌ Single brackets — breaks on empty variables, needs escaping
if [ $status = "active" -a $count -gt 0 ]; then
# ✅ Double brackets — safe with empty vars, supports && ||
if [[ "$status" == "active" && "$count" -gt 0 ]]; thenCommon conditional patterns:
# File checks
[[ -f "$file" ]] # File exists and is a regular file
[[ -d "$dir" ]] # Directory exists
[[ -x "$script" ]] # File is executable
[[ -s "$file" ]] # File exists and is not empty
# String checks
[[ -z "$var" ]] # String is empty
[[ -n "$var" ]] # String is not empty
[[ "$var" == *.log ]] # Glob pattern match
# Default values
name="${1:-anonymous}" # Default if unset
db_host="${DB_HOST:?'DB_HOST must be set'}" # Error if unsetThe :? syntax is invaluable for required environment variables. Instead of silently using an empty string, the script exits with a clear error message.
Processing Data Safely
Avoid parsing ls output or relying on word splitting to iterate over files. Use globs and find with proper null-delimiter handling.
# ❌ Parsing ls — breaks on spaces, special characters, symlinks
for file in $(ls /data/*.csv); do
process "$file"
done
# ✅ Glob pattern — handles all filenames correctly
for file in /data/*.csv; do
[[ -f "$file" ]] || continue
process "$file"
done
# ✅ find with null delimiter — recursive, handles everything
while IFS= read -r -d '' file; do
process "$file"
done < <(find /data -name '*.csv' -type f -print0)For JSON processing, use jq. For structured text, use awk. Avoid chaining grep | sed | cut for anything that a single jq or awk command could handle.
# ❌ Fragile pipeline — breaks if JSON format changes
curl -s "$API_URL" | grep '"name"' | sed 's/.*: "//;s/".*//'
# ✅ Structured JSON parsing with jq
curl -s "$API_URL" | jq -r '.items[].name'Portable Script Patterns
When a script needs to run across macOS, Linux, and CI environments, avoid platform-specific assumptions.
# ❌ GNU-specific flags — fails on macOS
date -d "2020-01-01" +%s
sed -i 's/old/new/g' file.txt
# ✅ Cross-platform alternatives
# Date parsing — use Python for portability
python3 -c "from datetime import datetime; print(int(datetime(2020,1,1).timestamp()))"
# In-place sed — macOS requires backup extension
if [[ "$(uname)" == "Darwin" ]]; then
sed -i '' 's/old/new/g' file.txt
else
sed -i 's/old/new/g' file.txt
fiWhen cross-platform compatibility becomes painful, it is a signal to switch to Python or Node.js. Shell scripting excels at orchestrating other commands. It struggles with string manipulation, data structures, and complex logic.
Key Takeaways
- Always use
set -euo pipefail— it prevents silent failures that cause real outages - Quote every variable —
"$var"is correct,$varis a bug waiting to happen - Use
[[ ]]not[ ]— double brackets handle edge cases that single brackets do not - Structure with functions and
main— even small scripts benefit from clear organization - Use
jqfor JSON, globs for files — avoid brittlegrep | sed | cutpipelines - Know when to stop — if the script needs data structures or error handling beyond try/catch, switch to a real programming language
