Transposes of apply / entries — ambient, corewise, and tangent#

If you need the transpose / adjoint / “back-projection” of apply or entries — for a gradient, a Gauss–Newton solve, or to build a tensor from measurements — read this first. There are three different transposes. They are not interchangeable, they return different things, and the one most people reach for by default (the corewise gradient) is deliberately not given the unqualified name. This doc explains what each one is, which to use, and what each costs. For the forward operations see entries_apply_probe.md; for the sum_over_probes stacking mechanics see batching_and_stacking.md §11.

Math reference: §6.2.3 (“Transpose of tangent vector to probes map”, Algorithm 8) and §6.3 (“Corewise (non-manifold) simplification”) of Alger, Christierson, Chen & Ghattas (2026), “Tucker Tensor Train Taylor Series” (arXiv:2603.21141). (As elsewhere, the local t4s.pdf numbering is newer than the public arXiv.)

TL;DR — three transposes, one picture#

The forward apply is linear in the tensor X: apply(X) = ⟨X, w₀ ⊗ ⋯ ⊗ w_{d-1}⟩. Its honest adjoint sends a residual c to the rank-1 tensor

c · (w₀ ⊗ w₁ ⊗ ⋯ ⊗ w_{d-1})            # the "ambient back-projection"

The other two transposes are that same object, projected onto a smaller space:

transpose

adjoint of …

returns

base point?

reach for it when …

ambient — TuckerTensorTrain.apply_ambient_transpose(c, ww)

apply on the full tensor space

raw tuple of canonical (CP) factors (c·⊗w)

no (static)

you want the literal back-projection: a building block, a rank-1 update, to combine back-projections across base points, or to hand off to a CP/general-tensor library

corewise — X.apply_corewise_transpose(c, ww)

the core parametrization at X

raw tuple (tucker_grads, tt_grads)

yes (self)

you optimize the cores directly with a core-wise optimizer (Adam, L-BFGS, plain SGD)

tangent — T3Tangent.apply_transpose(c, ww, frame)

the manifold tangent map at X

a T3Tangent

yes (orthogonal, gauged)

you do Riemannian optimization on the fixed-rank manifold

The relationships:

  • corewise = the ambient back-projection projected onto the core-parametrized tangent space at X (the span of single-core perturbations).

  • tangent = the ambient back-projection projected onto the gauged manifold tangent space at X.

Everything below is identical for entries (entries_ambient_transpose, entries_corewise_transpose, T3Tangent.entries_transpose) and for probe (probe_ambient_transpose, probe_corewise_transpose, T3Tangent.probe_transpose) — all three sampling operations have all three transposes. probe has one structural twist; see Probe transposes below.

Why the explicit names (ambient vs corewise)#

Most users — reasonably — expect “the transpose of apply” to be the gradient of a loss with respect to the cores: you have cores, you computed a measurement, you want ∂loss/∂cores. That is the corewise transpose. But it is not the literal adjoint of apply:

  • apply is the linear map X ↦ ⟨X, ⊗w⟩. Its adjoint, in the textbook sense ⟨Aᵀc, X⟩ = ⟨c, A X⟩, is c·⊗w — frame-free, representation-free. That is the ambient transpose.

  • The corewise gradient is the adjoint of a different (composite, frame-dependent) map cores ↦ measurement.

If apply_transpose were unqualified it would have to mean one of these, and whichever we picked, half of all users would silently get the wrong object. So both carry an explicit qualifier. The static-vs-instance split reinforces it: the ambient transpose is frame-free (a @staticmethod — it depends on nothing but c and the probe vectors), while the corewise transpose differentiates your cores (an instance method, self is the base point).

The three, in detail#

Ambient — apply_ambient_transpose(c, ww, sum_over_probes=…) (static)#

The literal adjoint of apply, returned as a raw tuple of canonical (CP) factors — len = d, one factor per mode, c folded into the first. It depends on nothing but c and the probe vectors; there is no base point.

CP is the natural type here, by duality: apply consumes one vector per mode and returns a scalar; its adjoint takes a scalar and returns one (scaled) vector per mode. In CP form the operation is almost a relabelling —

apply_ambient_transpose(c, ww)  =  (c⊙w₀, w₁, …, w_{d-1})        # CP factors, c folded into factor 0

— which is exactly what an honest adjoint of a multilinear form should look like. (Wrapping this in a TuckerTensorTrain would add an identity copy-tensor as the TT cores — pure structure, no information.)

  • Return convention: factors match what from_canonical consumes — len = d, factor[i].shape = stack_shape + (R, Nᵢ). So from_canonical(apply_ambient_transpose(c, ww)) round-trips to the back-projection as a TuckerTensorTrain, if you want it in T3 form (see Cost for when that conversion is expensive).

  • Defining property (and how you test it): once realized as a tensor, ⟨from_canonical(apply_ambient_transpose(c, ww)), X⟩ == c · X.apply(ww) for every X.

  • sum_over_probes picks where the probe stack W lands — and both are cheap in CP:

    • False (default): W is a stacking axis; the result is a W-stack of rank-1 CP tensors (R = 1) — one back-projection per probe.

    • True: W becomes the CP rank; the result is one rank-|W| CP tensor Σ_W c_W (⊗w^W), stored in O(d·|W|·N) (see Cost — no blow-up, because CP keeps the shared rank index implicit).

  • Composable / interop. It lives in the full ambient tensor space, so you can add ambient back-projections from different base points or iterations, project onto any tangent space, etc. And because it’s a plain factor tuple — not a bespoke class — it hands off directly to dedicated CP / general-tensor libraries. (This library is deliberately laser-focused on T3, not a general CP library; the raw tuple is the clean boundary.)

  • entries variant (entries_ambient_transpose(c, index, shape, sum_over_probes=…)): the CP factors are the one-hot vectors e_{indexᵢ}, fully determined by the indices (shape supplies the ambient dims, which c and index alone don’t). Same semantics otherwise.

Corewise — X.apply_corewise_transpose(c, ww, sum_over_probes=…) (instance)#

The gradient of the measurement with respect to X’s cores, treated as independent optimization variables. Returns a raw tuple (tucker_grads, tt_grads) whose arrays have the exact shapes of X’s cores — so each entry maps one-to-one onto a core you are optimizing.

  • This is the operation behind standard (non-manifold) optimizers: cores ← cores − lr · grad.

  • It is implemented by the §6.3 substitution into the tangent machinery (Algorithm 8 with the TT cores used in place of the orthogonal left/right/down frames, and the Tucker cores no longer required to be orthogonal). You do not need an orthogonal representation to call it.

  • Returns a raw tuple, not a TuckerTensorTrain, on purpose: the result is a gradient, not a tensor. Wrapping it in a TuckerTensorTrain would invite X − lr·grad, which silently does tensor-space arithmetic (rank concatenation), not the core-wise step you want.

  • Caveat (from §6.3): the cores are a representation, not an intrinsic object, so a corewise gradient is meaningful only at the base point where it was taken. Combining corewise gradients from different base points (as L-BFGS history does) is, strictly, ill-defined from the manifold viewpoint. Core-wise optimizers do it anyway and it works in practice; just know that is what is happening. If you want the principled version, use the tangent transpose.

Tangent — T3Tangent.apply_transpose(c, ww, frame, sum_over_probes=…) (static, takes a frame)#

The Riemannian Jacobian transpose: back-projects c into a tangent vector at a base point on the fixed-rank manifold. Returns a T3Tangent. Requires an orthogonal (gauged) frame; the resulting tangent is the Riemannian gradient after the gauge projection. This is the operation for Riemannian optimization (Newton-CG, RGD); see the worked example examples/fit_hilbert_tensor_newton_cg.py. It lives on T3Tangent, so the class name already says “tangent” — no qualifier needed.

Cost — and why the representation matters#

The crucial fact is where the cost lives, and CP versus T3 puts it in different places.

  • Ambient (CP), both modes: O(d·|W|·N). Unsummed is a W-stack of rank-1 CP; summed is one rank-|W| CP. Summed is not more expensive, because CP keeps the shared rank index implicit — Σ_W c_W (⊗w^W) is just the factor matrices [wᵢ^W] ∈ ℝ^{|W|×Nᵢ} plus the weights c. Linear in |W|. This is the whole reason CP is the right output type, and why apply_ambient_transpose keeps sum_over_probes.

  • The |W|² cost is real, but it lives in from_canonical (CP → T3), not in the transpose. A dense Tucker tensor train of a rank-|W| object must materialize the “all factors share one index” coupling as a diagonal/copy core — ≥ |W|² storage (and there’s no exact, data-independent way around it: the minimal ranks are data-dependent, which would break stacking and give non-exact results). By returning CP, the transpose stays cheap and that conversion cost becomes the user’s explicit opt-in — incurred only if and when they actually convert to T3, which is exactly where it belongs.

  • Corewise / tangent, summed (sum_over_probes=True): the gradient is shaped like the cores (corewise) or the tangent space (tangent) — a fixed size set by X, not by |W|. Summing over W collapses the |W| rank-1 contributions into those fixed-shape objects. No |W| ever appears in a rank. sum_over_probes=True here is the Gauss–Newton Jᵀr, and it is cheap.

So for an optimization gradient, prefer corewise/tangent (fixed-size, fused). The summed ambient CP is the honest “ambient Euclidean gradient as a tensor” — cheap to form, but to use it (project, fit) you’d pay O(|W|) anyway, so it’s mainly for the general/unpredictable case, sketching, or interop, not the optimization hot path.

The forward/transpose N-asymmetry (especially for entries)#

Forward entries is N-free: it reads X[i] by slicing the cores at the index, so the ambient mode dimension N never enters the arithmetic. The transposes are not N-free — but that is intrinsic to the output, not a flaw of the construction. Their outputs are tensors / tangents with N-dimensional modes, so emitting the result is unavoidably Ω(N). The forward gets to skip N because its output (a scalar) contains no N; the transpose’s output does. (Inside a Riemannian solve even this is sidestepped: you never need the standalone back-projection, only its inner products against tangent directions, and those fuse back to N-free slicing by adjointness.)

Which one do I want?#

Do you want the back-projected TENSOR itself (a building block, a rank-1
update, something to combine across base points, or to hand to a CP library)?
        └── yes → apply_ambient_transpose            (raw CP factor tuple; → from_canonical for a T3)

Are you optimizing the cores directly with Adam / L-BFGS / SGD?
        └── yes → X.apply_corewise_transpose         (raw (tucker_grads, tt_grads))

Are you doing Riemannian optimization on the fixed-rank manifold?
        └── yes → T3Tangent.apply_transpose          (a T3Tangent)

All three have entries and probe counterparts with the same semantics, and all three take sum_over_probes (True = sum the probe stack W; False = keep it as a stack, one result per probe — useful for assembling JᵀJ matrix-vector products). The difference is where the sum lands: cheap in all three (CP rank for ambient; the fixed-size cores/tangent for corewise/tangent), but only corewise/tangent keep the result at a fixed size independent of |W|.

Probe transposes (the one structural twist)#

probe contracts all-but-one mode, returning d vectors (one free mode each) instead of a scalar. So its transposes take a residual that is d vectors ztildes = (ž₀,…,ž_{d-1}) (plus the probe vectors ww), where apply/entries took a scalar residual c. Everything else is the same three transposes:

  • probe_ambient_transpose(ztildes, ww) (static) — the literal adjoint of probe. The back-projection is Σᵢ (w₀ ⊗ … ⊗ žᵢ ⊗ … ⊗ w_{d-1}) (residual žᵢ in slot i, probe vectors elsewhere) — a rank-d CP tensor (vs rank-1 for apply), returned as CP factors. Summed → rank d·|W|, still O(d|W|N).

  • X.probe_corewise_transpose(ztildes, ww) (instance) — the gradient of X.probe(ww) w.r.t. X’s cores, for non-manifold optimizers fitting from probes. Raw (tucker_grads, tt_grads), no |W| blow-up.

  • T3Tangent.probe_transpose — the Riemannian gradient (the paper’s original probing Jacobian transpose).

Probing is the paper’s exemplar operation, so this completes the full 3×3 grid (entries/apply/probe × ambient/corewise/tangent).