A Production-Quality Educational Unix Shell Implementation

Python Shell (psh) is a POSIX-style, bash-compatible shell written entirely in Python, designed for learning shell internals while providing practical functionality. It features a clean, readable codebase with modern architecture and powerful built-in analysis tools.

Current Version: 0.797.0 | Tests: 30,000+ | Compatibility: POSIX + bash, verified against live bash

All source code and documentation (except this note) has been written by Claude Code using Sonnet 4.x and Opus 4.x models.

# Install
git clone https://github.com/philipwilson/psh.git
cd psh && pip install -e .

# Run interactively
psh

# Execute commands
psh -c "echo 'Hello, World!'"

# Analyze scripts
psh --metrics script.sh
psh --security script.sh
psh --format script.sh

• 🔍 CLI Analysis Tools: Built-in script formatting, metrics, security analysis, and linting
• 📚 Educational Focus: Clean, readable codebase designed for learning shell internals
• 🧪 Comprehensive Testing: 30,000+ tests ensuring reliability and robustness
• 🏗️ Modern Architecture: Component-based design with unified lexer and visitor pattern integration
• 🎓 Dual Parser Implementation: Production recursive descent parser, plus an educational parser-combinator alternative for comparing parsing paradigms
• 📋 POSIX + bash compatible: behavior is conformance-tested against live bash (see the compatibility matrix in the user guide)
• 🎯 Feature Complete: Supports advanced shell programming with arrays, functions, and control structures

PSH includes powerful built-in tools for shell script analysis:

psh --format script.sh              # Format with consistent indentation
psh --format -c 'if test; then; fi' # Format command strings

psh --metrics script.sh             # Analyze complexity and code metrics
psh --security script.sh            # Detect security vulnerabilities  
psh --lint script.sh                # Style and best practice suggestions

Example Output (the script lives in examples/fibonacci.sh):

$ psh --metrics examples/fibonacci.sh
Script Metrics Summary:
═══════════════════════════════════════
Commands:
  Total Commands:            22
  Unique Commands:           10
  Built-in Commands:          4
  External Commands:          6

Structure:
  Functions Defined:          2
  Pipelines:                 0
  Loops:                     3
  Conditionals:              1

Complexity:
  Cyclomatic Complexity:      5
  Max Pipeline Length:        0
  Max Nesting Depth:          2
  Max Function Complex:       1

Runnable example scripts live in examples/ — see examples/README.md for the full set.

• Command Execution: External commands, built-ins, background processes (&)
• I/O Redirection: All standard forms (<, >, >>, 2>, 2>&1, <<<, <<)
• Pipelines: Full pipeline support with proper process management
• Variables: Environment and shell variables with full expansion
• Special Variables: $?, $$, $!, $#, $@, $*, $0, positional parameters

• Parameter Expansion: All bash forms including ${var:-default}, ${var/old/new}, ${var^^}
• Command Substitution: Both $(cmd) and `cmd` with nesting support
• Arithmetic Expansion: $((expr)) with full operator support and command substitution
• Brace Expansion: {a,b,c}, {1..10}, {a..z} with nesting
• Process Substitution: <(cmd) and >(cmd) for advanced I/O patterns
• Glob Expansion: *, ?, [abc], [a-z] with quote handling and extended globbing (extglob)

• Control Structures: if/then/else, while, for, case, C-style for ((;;))
• Functions: POSIX and bash syntax with local variables and return values
• Arrays: Both indexed and associative arrays with full bash compatibility
• Break/Continue: Multi-level loop control with break 2, continue 3
• Test Commands: Both [ and [[ with comprehensive operator support

• Line Editing: Vi and Emacs key bindings with customizable modes
• Tab Completion: Intelligent file/directory completion with special character handling
• Command History: Persistent history with search and navigation
• Job Control: Background jobs, suspension (Ctrl-Z), jobs, fg, bg
• Prompt Customization: PS1/PS2 with escape sequences and ANSI colors

All 61 registered builtins (from psh.builtins.registry):

Core: cd, pwd, echo, exit, true, false, :, exec
Directory Stack: pushd, popd, dirs
Variables: export, unset, set, shift, declare, typeset, local, readonly, let, getopts, env
I/O: read (with -p, -s, -t, -n, -d options), printf, print, mapfile (alias readarray)
Job Control: jobs, fg, bg, wait, kill, disown
Functions & Execution: return, source, ., eval, command, builtin, type, hash
Signals: trap (signal traps plus EXIT, DEBUG, ERR)
Testing: test, [
Shell Options & Environment: shopt, umask, times
History & Aliases: history, alias, unalias
PSH Introspection: help, version, signals, parser-select, parser-mode, parser-config, parse-tree, show-ast, ast-dot, debug, debug-ast

# Commands and pipelines
ls -la | grep python | wc -l
find . -name "*.py" | xargs grep "TODO"

# Variables and expansions
name="World"
echo "Hello, ${name}!"
echo "Current directory: $(pwd)"
echo "2 + 2 = $((2 + 2))"

# Functions with local variables
calculate() {
    local a=$1 b=$2
    echo $((a * b))
}

# Arrays and iteration
files=(*.txt)
for file in "${files[@]}"; do
    echo "Processing: $file"
done

# Associative arrays
declare -A config
config[host]="localhost"
config[port]="8080"
echo "Server: ${config[host]}:${config[port]}"

# Enhanced conditionals
if [[ $file =~ \.py$ && -f $file ]]; then
    echo "Python file found"
fi

# C-style loops
for ((i=0; i<10; i++)); do
    echo "Count: $i"
done

# Case statements
case $1 in
    start) echo "Starting service" ;;
    stop)  echo "Stopping service" ;;
    *)     echo "Usage: $0 {start|stop}" ;;
esac

# Format messy scripts
psh --format messy_script.sh > clean_script.sh

# Security analysis
psh --security deploy.sh
# Output: [HIGH] eval: Dynamic code execution - high risk of injection

# Code metrics for complexity analysis
psh --metrics complex_script.sh
# Shows cyclomatic complexity, nesting depth, command usage

# Linting for best practices
psh --lint old_script.sh
# Suggests modern alternatives and style improvements

Requires Python 3.12 or later.

# Clone and install
git clone https://github.com/philipwilson/psh.git
cd psh
pip install -e .

# Install development dependencies
pip install -e ".[dev]"

# Run tests
python -m pytest tests/

PSH follows a modern component-based architecture with clear separation of concerns:

• Shell (psh/shell.py): Main orchestrator coordinating all subsystems
• Lexer (psh/lexer/): Modular tokenization with mixin architecture
• Parser (psh/parser/): Dual parser implementation:
  • Recursive Descent (recursive_descent/): Production parser with modular package structure
  • Parser Combinator (combinators/): Educational functional-parsing alternative (not production-supported)
• Executor (psh/executor/): Command execution with specialized handlers
• Expansion (psh/expansion/): All shell expansions with proper precedence
• I/O Management (psh/io_redirect/): File operations and redirection handling
• Interactive (psh/interactive/): REPL, completion, history, and prompts

PSH implements the visitor pattern for AST operations, enabling:

• Analysis Tools: Metrics, security scanning, linting
• Code Transformation: Formatting, optimization
• Extensibility: Easy addition of new analysis features

PSH includes two parser implementations with deliberately different statuses:

• Recursive Descent Parser: The production parser — modular package structure, clear error messages, comprehensive shell support. All conformance and correctness work targets this parser.
• Parser Combinator: An educational alternative demonstrating functional composition. It handles the broad shell grammar and is pinned against drift by parity tests, but it is outside the production quality bar: it may lag on edge cases (known gaps include composite words in some list contexts) and its gaps are not tracked as defects.
• Educational Value: Compare and contrast imperative vs. functional parsing approaches
• Parser Selection: Use parser-select combinator builtin (or --parser combinator) to switch implementations interactively

• Lines of Code: ~84,506 lines of production code in psh/ across 276 Python files, plus ~171,013 lines of tests in tests/ (839 Python files)
• Test Coverage: 30,379 tests in 864 test files
• Architecture: 8 major components with focused responsibilities
• Visitors: 7 analysis and transformation visitors (psh/visitor/)
• Dual Parser: Both recursive descent and parser combinator implementations

Canonical testing commands for contributors and CI are maintained in docs/testing_source_of_truth.md.

Use the provided test runner for correct handling of all tests:

Three validation tiers (named the same here, in run_tests.py, and in docs/testing_source_of_truth.md):

# standard tier — THE local gate (whole suite, parallel + serial + subshells)
python run_tests.py --parallel

# quick tier — curated smoke subset (unit + fast integration), parallel, ~20s
python run_tests.py --quick

# smart serial mode (same phases as the gate, without xdist)
python run_tests.py

# all tests with capture disabled (simpler but noisy)
python run_tests.py --all-nocapture

The quick tier is for fast local iteration and is not sufficient to merge; the standard --parallel run is the gate. A nightly workflow runs the full tier (standard plus live-bash conformance and coverage) on Linux as a backstop — see the testing source of truth for the exact CI contract.

# All tests - run normally (subshell tests no longer need -s, as of v0.195.0)
python -m pytest tests/

# Subshell tests
python -m pytest tests/integration/subshells/

# Specific categories
python -m pytest tests/unit/           # Unit tests
python -m pytest tests/integration/    # Integration tests
python -m pytest tests/conformance/    # POSIX/bash compatibility

# Performance tests
python -m pytest tests/performance/

# Coverage reporting
python -m pytest tests/ --cov=psh --cov-report=html

Note: As of v0.195.0 the full suite passes under normal pytest capture; the -s flag is no longer required for subshell tests (a read builtin fix made it read the real redirected file descriptor). run_tests.py still works and remains the recommended runner.

Current test status: the full suite (see the Test Coverage count above) passes locally via python run_tests.py --parallel, with a few hundred tests skipped as platform-specific or interactive. Run the command for exact pass/skip totals on your platform — see docs/testing_source_of_truth.md.

PSH targets POSIX shell semantics with a large set of bash extensions. The project does not publish a single compliance percentage — there is no mechanical POSIX oracle behind such a number. Instead, compatibility is established by conformance tests that compare PSH against live bash on the same host (tests/conformance/, the shared oracle runner's resolve_bash() picks the newest available bash). POSIX-scoped cases live under tests/conformance/posix/ and bash-extension cases under tests/conformance/bash/; each asserts identical stdout/stderr/exit status or a catalogued, documented difference.

The authoritative, per-feature picture is the compatibility table in docs/user_guide/17_differences_from_bash.md, which marks every feature Full / Partial / No and is kept honest by a meta-test (tests/conformance/test_claims_have_tests.py) that requires a proving conformance test for each "Full support" claim.

• ✅ Shell Grammar: all major constructs (pipelines, lists, compound commands)
• ✅ Parameter Expansion: the standard ${...} forms plus bash string operators
• ✅ I/O Redirection: files, fds, heredocs, here-strings, process substitution
• ✅ Control Structures: if/while/until/for/case, C-style for, select
• ✅ Built-in Commands: the POSIX special builtins plus common bash builtins