Compare two HDF5 files — as an explorer with a difference overlay, not a text-style diff.

A changed .h5 file is invisible to normal tooling. git diff says only "Binary files differ." Dumping both files and diffing the text is worse than useless: one changed (1,2,16,16) complex array produces 51 KB of floating-point numbers, and the signal drowns.

The concrete case that motivated this: a 25 KB SAFIRE Hamiltonian file where the only change was one scalar string four levels deep, hst_type going discrete_spin → continuous_spin. Nothing in the existing toolchain could show that.

$ python -m h5diff_engine compare old.h5 new.h5
27 nodes: 1 differing · 26 equal · 0 only in A · 0 only in B

M Hamiltonian
M   ModelHamiltonian
M     ModelComponent_1
M       hst_type   scalar string(vlen, utf-8)  |  scalar string(vlen, ascii)
          A: discrete_spin
          B: continuous_spin

There is no comparison tool for HDF5 in this space. H5Web (excellent, MIT) views a single file, reading it with h5wasm inside the webview — which is also why it caps out at 2 GB.

100 tests pass: 91 Python (pytest) and 9 headless Node protocol tests. The Python suite includes a head-to-head cross-check that agrees with SAFIRE's own comparator on 182/182 real reference files.

Tolerance, preset, and NaN handling are settings (hdf5diff.*) and can also be changed live from the panel header, which re-runs the comparison.

┌─ Webview (client) ─────────────┐   postMessage    ┌─ Extension host (remote, Node) ─┐
│  vanilla TS + HTML/CSS         │ <──────────────> │  extension.ts                   │
│  union tree, detail pane,      │   JSON only      │  PythonWorker (spawn + NDJSON)  │
│  canvas heatmaps/line charts   │                  └────────────┬────────────────────┘
└────────────────────────────────┘                               │ stdin/stdout NDJSON
                                                    ┌────────────┴────────────────────┐
                                                    │  python -m h5diff_engine.server │
                                                    │  h5py + numpy. Holds both files │
                                                    │  open; streams stats.           │
                                                    └─────────────────────────────────┘

All HDF5 reading and all numerics happen in Python (h5py + numpy). The TypeScript side never opens a file. Two reasons:

• Remote workspaces. On VSCode remote the webview runs client-side while the files live on the remote filesystem. Reading in the extension host and shipping only summaries is what removes the size ceiling.
• Runtime dependencies. npm is confined to build tooling; there are no runtime npm dependencies at all.

The engine has no editor dependency — it is driven by the CLI and by pytest, so correctness is established without VSCode in the loop.

Entities are paired by path, because an HDF5 group is a keyed map: nothing "moves down three lines" the way text does. Each path gets one status, from the first rung that applies.

1. Presence — ONLY_A / ONLY_B. One-sided subtrees stay browsable.
2. Kind — group vs dataset vs link → KIND_MISMATCH.
3. Datatype — classify the relationship, not just inequality. Integer width drift (int32 ↔ int64), float32 ↔ float64, fixed-width bytes ↔ vlen utf-8, int ↔ float, real ↔ complex are all coerced and still compared, with a warning. Compound types pair fields by name. Numeric-vs-string is terminal.
4. Shape — alignment strategies, first match wins, and the choice is always reported: exact → squeeze (drops length-1 axes) → prefix (compares the overlap of two different-length runs) → transpose → flatten → overlap (opt-in) → none. When shapes cannot be aligned, scalar reductions still distinguish "same data, different layout" from "genuinely different data".
5. Values — numpy convention, |a-b| <= atol + rtol*|b|. NaN and Inf are first-class: nan_only_a, nan_only_b, both_nan, inf_mismatch are counted separately, because "this became NaN" is the most important diff in this domain.
6. Attributes — same ladder, one level down. An attribute-only change reports ATTR_DIFF.

Ambiguous shapes are resolved by evidence, not precedence: (4,6) against (6,4) could be a transpose or a relayout, so both are tried and whichever the data supports wins.

A differing dataset is described by n_diff/n_total, max_abs with its unravelled index, max_rel, l2, per-side extrema, NaN/Inf counts, and a log-spaced histogram of |diff|. The histogram is the highest-value widget in the tool — it separates "everything moved by 1e-16" (reordered reductions, harmless) from "a few entries moved by 1e-2" (real behavioural change) at a glance.

Datasets are read slab-by-slab along the leading axis of the aligned region, so peak memory is bounded regardless of dataset size.

export PYTHONPATH=$PWD/python

# text output; exit code 0 = identical, 1 = differences, 2 = error
python -m h5diff_engine compare a.h5 b.h5

# machine-readable, for tooling
python -m h5diff_engine compare a.h5 b.h5 --json

# tolerance control
python -m h5diff_engine compare a.h5 b.h5 --exact          # bitwise
python -m h5diff_engine compare a.h5 b.h5 --atol 1e-8

# show everything, not just differences
python -m h5diff_engine compare a.h5 b.h5 --all

# no domain knowledge: nothing ignored, no complex heuristics
python -m h5diff_engine compare a.h5 b.h5 --preset generic

Requires h5py and numpy only.

The engine is generic; project knowledge lives in presets.py as globs. The safire preset (default) mirrors choices already made in SAFIRE's run_functional.py:

• ignore — run_time_seconds, input_file. Both differ on every rerun. Ignored nodes are shown greyed, never hidden, so suppression is visible.
• explanatory — num_ranks. A rank-count mismatch explains every other difference in the file, so it is surfaced first with a banner.
• stream_only — DenseFactorized/L, Interaction/Vq*, Walkers/WalkerSet/walkers_*, per-block *_rdm_*. Never fully materialised.
• complex_trailing2 — paths where a bare trailing axis of 2 may be read as complex.

SAFIRE stores complex numbers three different ways, and the engine reconciles all of them so a file written by one path compares against a file written by another:

native and attr are unambiguous and always honoured. trailing2 is indistinguishable from a genuine (N,2) real array, so it is only honoured for paths explicitly opted in. When one side is native and the other attr, the __complex__ marker is not reported as a difference — it describes storage the comparison already normalised.

An interpreter recognises a convention layered on top of groups and datasets, and compares the thing the convention encodes rather than its storage. Each lives in python/h5diff_engine/interpreters/, is independently testable, and is switched on by config — so a project with different conventions supplies its own preset without touching the ladder. --no-interpreters turns them all off, which is the escape hatch when the interpretation itself is under suspicion.

A group holding dims, data_, jdata_, pointers_begin_ and pointers_end_ is one sparse matrix. Comparing those five datasets element by element is wrong: the same matrix can be stored with its within-row entries in any order, so a regenerated file reports dozens of differences while encoding the same operator.

So both sides are reconstructed as (row, col) → value and that is compared. The comparison is logical, not structural: a position stored explicitly as zero on one side and absent on the other is equal, because both mean "this entry is zero". A position stored on one side only with a nonzero value is a real difference. A changed entry is reported at its (row, col), not at a CSR offset — the offset is an artefact of the storage, the coordinate is what you can act on.

The member datasets stay listed (marked ignored) so the interpretation can be checked.

s0..s23, ModelComponent_0..N, Vq0..Vq7, and above all the per-block observables in *.stat.h5 — one one_rdm_%09d and one denominator_%09d per measurement block, thousands per Average_<i>. Two runs of different length otherwise produce hundreds of near-identical rows.

Siblings matching prefix<digits>suffix, at least family_threshold (default 4) of them, collapse into one expandable row reporting counts per side, the index range, and which members differ by name — otherwise you must expand the family and hunt, which defeats the collapse. Presentation only: every member is still compared.

On a realistic 1200-vs-950-block stat.h5 pair (7212 nodes):

input_file is a whole afqmc.json in one scalar. Compared as text it is all-or-nothing; parsed, it names the settings that changed (afqmc.execute.timestep: 0.005 → 0.01).

Structural comparison is attempted for any string scalar that parses as a JSON object or array — it reports the same difference a text comparison would, only localised to a key. Path normalisation (reducing embedded absolute paths to bare filenames, so two checkouts of one case compare equal) is lossy, so it applies only to paths in the semantic_json list.

error_messages / warning_messages follow a {num_messages, error_i} layout. The index a message lands on carries no meaning, so they are compared as sets of decoded strings. Differences are reported but never fatal: the text describes a run, it is not the run's data.

Each side is labelled from its top-level structure — "model Hamiltonian", "CoQui k-point THC Hamiltonian", "NOMSD wavefunction", "estimator output (stat.h5)". Returns a list, because one file can be several things at once (rashba_soc/...free_elec_trial.h5 holds both a Hamiltonian and a wavefunction).

Node on this host is broken in two different ways and the failure is silent, so use the provided script rather than improvising:

. dev/env.sh          # loads node 22, provisions npm via corepack, sets PYTHONPATH
npm install
npm run compile

What env.sh works around:

• /usr/bin/node is v10.24.0 with npm 6.14.11 — far too old.
• Lmod node-js/22.14.0 gives a good node but ships no npm at all, and after loading it which npm still resolves to /usr/bin/npm. That is the trap.
• corepack enable cannot write shims into the module's bin directory (read-only nix store), so they go to ~/.npm-global/bin explicitly.
• npm is pinned to 10.9.2: npm 12 refuses node 22.14 (it wants ^22.22.2).

# engine -- the fastest loop, no editor involved
export PYTHONPATH=$PWD/python
python -m pytest python/h5diff_engine/tests/ -q
python fixtures/make_fixtures.py            # regenerate the HDF5 case pairs

# extension/engine boundary -- headless, no extension host needed
npm run compile && node out/test/protocol.test.js

Press F5 for the Extension Development Host. Test on a remote workspace specifically: that is the configuration where reading in the webview rather than the extension host would break, and it is what the architecture is built around.

test_ladder.py covers every rung against the generated fixture pairs, test_interpreters.py covers the domain interpreters, and test_real_files.py runs against a real SAFIRE checkout (skips cleanly without one) including the cross-check against run_functional.compare_exactly. Point SAFIRE_ROOT at a different checkout if needed.

Real-file tests assert properties rather than exact path lists: the results.h5 schema has already been reorganised once (the payload moved from flat energy/avg_1rdm into measurements/Stage0/...), and pinning paths would make them a tripwire for upstream refactors rather than for this tool.

Both gaps that Phases 1–3 left open are now closed by interpreters, and their fixtures remain as regression tests: csr_reorder (a reordered sparse matrix now compares equal) and family_blocks (120-vs-95 per-block datasets now collapse to two rows).

What is still missing:

• No charts. Differences are described numerically and with an ASCII histogram. Heatmaps, line charts and the matrix view are Phase 5.
• No git integration. Comparing against HEAD still means materialising the blob by hand (git show HEAD:path > /tmp/old.h5). Phase 6.
• No statistical mode. Stochastic observables are compared against a tolerance, not against their own error bars. The new schema stores every observable as a {mean, error} group, which is the natural hook. Phase 8.
• Family collapsing only handles numbered siblings. The measurements/ schema uses named ones (PairCorr/+x_+y, Steps=2), which are left as individual rows.
• Speed. A 7212-node stat.h5 pair takes ~7 s, dominated by comparing thousands of small datasets. Fine interactively; worth revisiting if files grow.