Testing strategy — correctness, and the clean-padding blind spot#

The why behind how this library is tested. The mechanics (unittest + subTest, numpy-only numerical correctness with jax dispatch in tests/test_dispatch.py, doctests as reproducible examples) are in CLAUDE.md and doctest_style.md. This note records the deeper reasoning — in particular why dense/numerical tests alone are not enough for a representation that carries explicit structural metadata (the uniform layer’s rank masks), and the two tools that close the gap. The hardest-won lessons are from the uniform tangent layer; the principles are general.

The model: equivalence against ground truth#

Correctness is equivalence, checked two ways that reinforce each other:

  1. Dense ground truth. Rebuild the represented tensor with a hand-written np.einsum / .to_dense() and compare residual norms (~1e-12..1e-16). This is the bedrock — it does not trust any of the library’s own machinery.

  2. The cross-representation equivalence contract. The uniform layer is a faster ragged layer: to_ragged(uniform_op(to_uniform(x))) == ragged_op(x) on the real (masked) content, per stack element (garbage in the padding is don’t-care). The ragged layer is the trusted reference; every uniform op is verified against it. See uniform_equivalence_contract.md.

These two together are necessary. They are not sufficient.

The blind spot: too-permissive masks (phantom rank)#

A uniform object is supercores + boolean rank masks. The masks are structural metadata — they name the real rank (which stratum of the determinantal variety the point sits on). Two failure modes for a mask:

  • Too restrictive (drops real content): the dense is wrong → caught by ground-truth comparison. ✓

  • Too permissive (claims rank where the supercore is zero — phantom rank): the dense is still right (it kept zeros) → invisible to every dense/numerical test. ✗

Crucially, all our natural test inputs have clean (zeroed) padding — they come from ut3_orthogonal_representations / from_t3, which canonicalize. On clean padding, a phantom-rank mask is completely silent. This is the single most dangerous bug class here, because the wrong rank propagates: a downstream op that trusts the mask (SVD truncation in retract, sub-block extraction in to_t3, rank-controlled optimization) silently does the wrong thing.

This is not hypothetical. utv_to_ut3 once built the doubled-rank tensor with ones boundary masks (claiming rR+rL rank at a boundary whose supercore is rank-1-and-zeros). Every dense test passed at ~1e-16. Only re-deriving the construction from the paper (Appendix A.3.1, eqs 50–53) caught it; the fix is the honest zeros boundaries that respect the block structure.

The two tools that close it#

1. Exact output-mask assertions (non-circular)#

Assert the produced object’s masks exactly equal an independently-derived expected mask — derived from a different source than the implementation, so the test is not the code checking itself. Examples:

  • the doubled-rank masks vs a from-the-frame-ranks construction of the prefix-pair Tucker mask and the [Q | P] honest-boundary TT mask (the paper rule), not vs the impl’s own concatenation;

  • retract’s ranks vs the ragged retract’s ranks (the ground truth — note these can drop below the frame ranks when frame+tangent is genuinely lower-rank, which is correct, not a bug);

  • gauge output masks == input masks (gauge must preserve them); project masks == the frame gauge masks.

The tautology trap. Corrupting an output’s padding using the object’s own mask and re-densifying always passes — to_dense applies that same mask, so it is testing that to_dense applies the mask, not that the mask is right. Non-circularity requires an independent notion of the true real-region.

2. Garbage-padded inputs (mask-once robustness)#

The contract says padding is garbage-don’t-care, so a correct op must give the identical result on a garbage-padded input as on a clean one (tests/test_uniform_manifold.py’s _corrupt helper writes large garbage into the masked-out region). This catches two things at once:

  • ops that read raw padding instead of masking-once on entry;

  • for ops that build from raw supercores (utv_to_ut3 does not mask its inputs — it relies on the honest output mask), the garbage flows into the output’s input-derived padding, so a too-permissive output mask there leaks it → the dirty dense differs from the clean one → caught.

(The phantom boundary mask is a constructed-zero region, not input-derived, so input-garbage alone does not reach it — that one needs the exact-mask assertion. The two tools are complementary, not redundant.)

The stack matrix#

Batching/stacking is the most error-prone part of the library (batching_and_stacking.md). Every op test is parametrized over a standard matrix (_CONFIGS): the no-stack case, a frame (C) stack, a tangent (K) stack, K+C, forced-larger padding (so every core has a padded region, not just the sub-max-rank ones the default tight pad leaves clean), and multi-axis C=(2,3) / K=(2,3) (where the negative-axis / 1+|K| / suffix bookkeeping hides off-by-ones), plus a varying-rank-across-C stack (the rank-sweep case a single ragged object cannot even represent). Each axis targets a category that has bitten us before: stacking, masking, and TT-boundary handling.

Why the masks earned their keep#

A striking outcome of the hardening pass: once the tests were strict enough to see mask bugs, the implementation had none — exact masks, garbage robustness, multi-axis, varying-C, all green, confirmed independently by an adversarial cold-read audit. This is a point in favor of the masks-not-integer-ranks design (uniform_masks_vs_ranks.md, uniform_rank_masks_rationale.md). Boolean masks compose with the operations algebraically and locally: add = concat, multiply = Kronecker, the doubled-rank embedding = concat, gauge/projection preserve the mask, retraction’s SVD re-derives it. There is little integer rank-arithmetic to get wrong, and the few non-obvious spots (the [Q | P] doubled-bond order, the honest boundaries) are pinned by the exact-mask assertions above. A representation whose metadata travels through operations by concatenation and Kronecker products is simply harder to corrupt than one that threads hand-maintained integer ranks.

Exercising a mask check is not testing it — isolate it#

The two tools above tell you what to assert. This is about whether your assertion can actually fail, and it is the lesson of the weighted build (2026-07-15), where mutation testing caught a hole in the new tests twice — both times in a test that looked right, passed, and proved nothing.

The pattern to distrust: a test that constructs a bad object and asserts it is rejected. It passes. But which check rejected it?

  • S1. test_mask_mismatch_is_rejected perturbed only the tucker mask. Deleting the tt-mask comparison from ut3_weights_consistent entirely — a predicate half-blind — left the suite green.

  • S3. The frame-weight version widened the supercores as well as perturbing the mask. So the shape check rejected it, and the mask comparison was never reached: deleting the mask comparison left the suite green.

Both were fixed by making the mismatch mask-only and family-complete: perturb one mask family at a time, with every supercore byte-identical to the good object (assert that), so nothing but the mask comparison can be doing the rejecting. Then each deletion dies.

Two rules follow, and they generalize past masks:

  1. A rejection test must isolate the check under test. If any earlier check could also reject your bad input, you are testing that earlier check. Make the input good in every respect except the one.

  2. Enumerate the families. A check over k mask families needs k mismatches, not one. “It rejects a bad mask” is not the claim; “it rejects a bad mask on each family” is.

Why this bites here specifically: dense-vs-ragged agreement cannot see any of it. A half-blind predicate computes identical numbers on every well-formed input — it only misbehaves on the malformed input the test was supposed to construct and didn’t. That is the same blind spot as phantom rank, one level up: the checks need the same suspicion as the masks.

Mutation testing is the cheap way to find this. Delete the check, run the suite; if it stays green, the test is decorative. It is worth doing routinely on any mask-carrying op — it took minutes and found two real holes that review had not.

The adversarial audit pattern#

For subtle correctness work, a useful cross-check is an independent cold-read audit: a fresh agent with no shared assumptions reads the implementation against the math and the trusted reference and tries to construct a passing-but-wrong case (the verb is construct, not summarize). It is most valuable exactly where the author’s own mental model might hide a blind spot — and it is reassuring when it converges, independently, on the same suspicious surface the author targeted (here: the garbage-in-padding coverage gap) and finds it sound.

Checklist for a new uniform op#

When adding a uniform operation, write:

  1. dense-vs-ragged per stack element, over the _CONFIGS matrix (incl. forced-pad, multi-axis, varying-C);

  2. exact output-mask assertions against an independently-derived expectation (or, where the ground truth is the ragged op, against its ranks);

  3. garbage-padded-input robustness (clean == dirty);

  4. jax dispatch in tests/test_dispatch.py (jit the op; a stray np.* on a tracer raises), unless an in-file jit test already covers it;

  5. if the op has a structural precondition, a rejection test that isolates it (above) — and a quick mutation check that deleting the check turns the suite red.

Numbers 2 and 3 are the ones that distinguish a correct uniform object from a merely numerically plausible one. Do not skip them.