Ternary weights cost quality. The question is where the information goes instead.

The hypothesis, stated so it can lose

Ternarization flattens the network's internal representations. Where a full-precision layer has a few large activations carrying most of the signal, a ternary layer's activations are more uniform. That loss of contrast is the mechanism behind the quality drop. If so, mechanisms that restore contrast — a state-based attention with per-channel gating (KDA), a state-space mixer (Mamba-2), Qwen's fixed-size memory paired with block-indexed sparse retrieval, compressed long-range attention, a widened or manifold-constrained residual stream, or an attention over depth itself — should recover quality, and the recovery should be visible in the statistics, not just in the loss curve.

BitNet b1.58 constrains every weight to \( \{-1, 0, +1\} \) — about 1.58 bits each instead of sixteen, so a weight costs roughly a tenth of a byte to store. The published recipe leaves one slot conspicuously untouched: BitNet still runs full softmax attention with RoPE, and its own technical report names “investigating efficient attention mechanisms suitable for low-bit models” as future work. BitFuse is that investigation — but it borrows only the ternary technique: it quantizes the FFN/SwiGLU weight matrices alone (MLP-only) and keeps attention, embeddings, the tied head and every sequence mixer full-precision Qwen, so it is not a BitNet reproduction. And the newest literature keeps strengthening the original thesis rather than replacing it: Qwen, DeepSeek and Kimi have each shipped a different answer to “which dynamic carrier should do the work”, at full precision, without anyone testing whether the ranking survives ternarization.

Six carriers, one of them crippled

It helps to treat the network as a set of distinct information carriers, and to notice that ternarization is not symmetric across them. Ternarization is primarily a reduction in per-weight resolution, not a total loss of magnitude information — the model keeps a full-precision latent parameter and derives its ternary forward weight through a scale factor. So the useful question is not “how much accuracy is lost?” but “which channels become more valuable when static weight resolution is constrained?”

That asymmetry is the opening. A ternary transformation cannot express fine distinctions in its weights, but the dynamic carriers can still create sharp differences in what each token reads, remembers and preserves. If a low-precision network is short of contrast, those are where it can still be manufactured — and the organizing principle of the whole programme follows: compress static weight expressivity, measure where the information moves, then reinvest the freed budget in whichever mechanism delivers the most quality per unit of memory or compute.

What “controlled” means here, enforced in code

Every model is randomly initialized. The pretrained Qwen checkpoint is used only to confirm the architecture and tokenizer load correctly — never as a starting point. Held identical across all runs: tokenizer, dataset and revision, data ordering, token budget, sequence length, optimizer, warmup, gradient clipping, validation set, the seed set (42, 43, aggregated as the mean with the spread reported), checkpoint policy, evaluation protocol, hardware class.

That is not enforced by discipline. A fairness audit loads each variant's run manifest and compares every controlled field against an allow-list of the switches that are the experiment; any mismatch marks the whole comparison NON-CONTROLLED_EXPERIMENT and the reporting CLI exits non-zero, so an invalid comparison cannot pass silently in automation. Comparing the full model config against an allow-list — rather than a hand-maintained list of things to check — is what catches a hidden change like rms_norm_eps drifting on one variant.

The training recipe is unified. fp_baseline and every ternary variant use Qwen's plain cosine LR with constant weight decay, so fp_baseline → ternary_mlp carries no declared schedule difference — the MLP-only FFN weight precision is the only change. (A two-stage cosine schedule remains available as a possible future ablation of the BitNet b1.58 recipe, but no variant in the study uses it.) When a per-variant difference is intentional, the audit records it explicitly in a declared_differences list instead of waiving it silently — and only schedule fields may ever be declared. Declaring an unequal token budget raises, because that is not a recipe difference, it is a different experiment.

The shared core budget — embeddings, the per-layer mixer sized to the attention slot, the uniform SwiGLU MLP and the norms — is held fixed to within ~0.1% on the primary chain at the live 0.6B scale, never shrunk to compensate; whatever a mechanism adds beyond that core is reported transparently as its additional budget rather than hidden. Two families sit a little wider and are reported as-is rather than compensated away — the Kimi K3 family at −1.0% (its low-rank Gated MLA lands under the shared budget) and the KDA×QSA cross-family at +2.6% (its gated sparse-attention slot is heavier). That matching took deliberate tuning: KDA has no grouped-query KV saving and naive settings inflated it by 25%, a Mamba-2 mixer has an entirely different footprint, and the residual's full gated mode would have added ~29M parameters (+4.9%) — which is why the default is channel_gated at 2048 parameters per layer. Without that, “KDA is better” could just be “KDA is bigger”. Secondary matched-compute comparisons may relax the parameter rule, but they have to be labelled as such.

Four rules keep the expansion sets honest, and each one exists because a post-core mechanism would otherwise contaminate the primary table. MTP changes the training objective, so it is a separate Phase 5 extension set and is never merged into the Phase 2–4 perplexity comparison. The sparse MoE FFN changes both the training objective — a load-balancing auxiliary loss — and the parameter accounting, so it is compared on its active top-2 count with stored expert capacity reported separately. Multimodal runs report vision encoder and connector parameters separately, with language-backbone comparisons kept matched wherever that is possible at all. And every residual family arrives with a declared capacity budget, paid for outside the residual dimensions before the certified run.

Everything else holds as before: the same FineWeb-Edu revision, split, tokenization, shuffle seed and packing policy; the same optimizer family and schedule unless the experiment is an optimization study; the same precision, CUDA stack, kernel policy and evaluation environment; the same fixed cached validation set and cadence. Fake-quantized ternary measurements stay separate from genuine low-bit kernel measurements. Provenance recorded per run: git SHA, config hash, data revision, hardware, token count, checkpoint lineage, kernel versions, and every eviction and resume.

What this is not

• Not a speed claim. Ternary weights here are simulated (fake-quantized), which is slower than fp16, not faster. Efficiency reports are labelled algorithmic vs kernel_optimized and the two are never mixed.
• Not a state-of-the-art attempt. Optimizing for the best number would defeat the purpose; each architectural change has to stay attributable.
• MoE stays a Phase-6 ablation. The sparse MoE FFN (Q, R) is wired but deliberately sits behind the dense comparison, precisely to preserve attribution: learn which dense dynamic mechanisms are worth having before asking whether ternary savings are better spent on experts.
• Not a reproduction of anyone's frontier model. Mechanisms are borrowed from Qwen3.8, DeepSeek-V4.1-Flash and Kimi K3 one family at a time — the single deliberate exception being the KDA×QSA cross-family (XF-1, XF-2), which pairs Kimi's recurrent memory with Qwen's sparse retrieval as its own controlled comparison. Their fine-grained expert routing, their FP8/FP4 storage and their full multimodal stacks stay out — the Phase-6 MoE here is a plain top-2 Switch-style router, not their expert system.
• Not a claim that these effects are already reproduced. Everything attributed to an external paper is a literature input motivating an experiment, and is labelled as a proposal until BitFuse has run it.

Two bugs that argue for the whole methodology

Both were in the KDA kernel, both produced plausible-looking output, and neither was visible from a loss curve. The state decay was applied to the wrong axis — the state is \([d_v, d_k]\) and the gate indexes \(d_k\). And the intra-chunk term needs \(\exp(c_t - c_s)\) over cumulative log-decay; computing it as k / exp(c) overflows for long chunks or steep decay, so it is now split around the chunk midpoint to keep both exponential factors bounded by 1.

A chunk-parallel training kernel is asserted equal to an explicit recurrent reference to 2e-4 across chunk sizes, sequence lengths, initial states and streaming continuation. Without that reference to diff against, both bugs would have silently degraded models C and D and been misread as “KDA doesn't help.”

A third one is worth naming because it is invisible by construction: overriding _init_weights without calling super() leaves the RoPE inv_freq buffers zero-filled after from_pretrained, because they are non-persistent and absent from checkpoints. A resumed model then produces different hidden states from the one that was saved — identical weights, no error. It surfaced only as a save/load roundtrip test failing at ~1e-3. Every resumed training run would have been silently corrupt.

What gets measured

The headline number is not accuracy. It is marginal return per unit of resource, computed separately for each allocation axis:

where \(q\) is task quality and \(R\) is spent resource — weight memory, KV-cache, latency, training FLOPs, or energy where it can be measured honestly.

Alongside that, every run is instrumented internally, because a result without a mechanism is not worth much: ternary distribution and zero fraction per layer, activation magnitude, variance, contrast and effective rank, attention entropy, concentration, span and head diversity, recurrent-state occupancy, and how much earlier-layer information the residual pathway is actually carrying. The instrumentation is emitted at three levels — light, standard, full — and Phase 1's job includes confirming all three hold finite values before anything commits to a 1B-token run.

Each new mechanism brings its own diagnostics, each chosen so its mechanism could fail visibly rather than quietly. Weights gain layer-wise quantization error and scale drift. Attention gains QSA and CSA2 indexer recall and coverage, selected-block mass, and compressed-pool visibility statistics. Recurrent state gains GDN and KDA retention-and-forgetting curves. Residuals gain GR branch usage, mHC mixing entropy and deviation from double stochasticity, and AttnRes depth weights. MTP gains auxiliary-token losses, token acceptance rate and speculative-decoding speedup. The MoE FFN gains router load-balancing loss and expert-utilization histograms. Multimodal runs gain cross-modal retrieval and probe accuracy, image-token saliency, and modality-conditioned norm and entropy profiles.

A research-grade telemetry layer sits underneath all of it. Every run also records the per-step optimization trajectory and loss/gradient stability, a fixed probe set, and the internals of each moving part — MoE routing, the MTP heads, the hybrid mixers, attention logits, and the residual and gradient paths — plus efficiency benchmarks and post-hoc interaction, component-attribution, information-flow and stress-test analyses. It exists so a quality difference can be traced to a mechanism rather than asserted.

Reading the ternary weight carrier, straight from the checkpoint

The weight carrier gets its own static report — computed from the saved {−1, 0, +1} weights alone, with no data and no forward pass, so it can be recomputed from any checkpoint (including the Phase-0 ones already on disk, with no retraining) and kept as a lightweight result long after the full weights are reclaimed. One duck-typed interface — a module that exposes quantized_weight() — discovers the ternary layers in every family, so fp_baseline simply reports is_ternary: false and each new mechanism is picked up automatically. Every number is tagged static or dynamic, so its provenance — the weights alone versus a forward pass — is never ambiguous.

For every ternary tensor it records the full distribution — zero / positive / negative density, the positive and negative fraction among the non-zeros, and a sign balance in \([-1, +1]\) (\(+1\) and \(-1\) are never assumed equiprobable) — plus the ternary entropy in bits (max \(\log_2 3 \approx 1.585\)), the sign entropy, the per-tensor scale statistics, and that same distribution pooled per parameter, per module category and per decoder depth, so “which layers went sparse, and where” is answerable rather than averaged away. It is the evidence for the study's core question — where information relocates once ternarization removes the weights' precision — read from the weight channel itself.

Resumable training, bounded storage

A campaign this size — the thirty-six variants, more once scale profiles and budget sweeps are counted — cannot keep every finished model's full weights on disk. Training is fully resumable: each checkpoint stores the complete state (model, optimizer, scheduler, AMP scaler, token/step counters, the data-stream cursor and all RNG), written atomically with a checksummed manifest, so a crash, restart or Spot eviction continues exactly rather than restarting from zero — and every checkpoint is stamped with its experiment_id, so one run can never resume from another's state. But a run is marked COMPLETED only once training and metric collection have finished, and only then does it persist its lightweight ternary results and reclaim the large weight payloads, keeping the provenance sidecars and a cleanup.json marker. So disk grows with the number of running experiments, not finished ones — a fixed ceiling of one latest + one best + a bounded periodic history per active run — while the final results never depend on retaining the trained weights. Each run advertises its lifecycle — RUNNING, RESUMABLE, COMPLETED, FAILED — and a completed one is skipped, never silently retrained.

Four outcomes, all of them informative

• Ternary scaling wins. Dynamic capacity is not an efficient substitute for parameters at low precision — which retires a plausible idea and tightens the case for current practice.
• Allocation wins. A smaller ternary backbone with richer retrieval, residual or predictive machinery beats a larger one at matched resources, and low-bit scaling has been leaving quality on the table.
• They are complementary. Both axes help, and the useful contribution becomes the shape of the joint frontier.
• Allocation is task-dependent. Short context favors weights, long context favors selective retrieval, reasoning favors conditional capacity — the most interesting outcome, because it means there is no single correct low-bit architecture, only a correct allocation for a workload.

What counts as a strong result

A strong result is emphatically not “the newest architecture wins.” It is a controlled trade-off that survives matched-budget comparison and has a plausible mechanistic explanation behind it. So the claims are pre-committed with their interpretations attached:

• If Qwen GDN + QSA wins, the claim is about the usefulness of fixed-size memory plus selective retrieval under ternary weights — not about Qwen being superior.
• If mHC beats Gated Residual, the result is that constrained multi-stream routing offers a better stability/expressivity trade than unconstrained gating.
• If AttnRes wins, the contribution is evidence that depth-wise information selection is a carrier separate from parallel residual streams.
• If MTP improves serving but not validation loss, that is still a meaningful efficiency result, and the paper labels it as exactly that.
• If multimodal performance improves only with a large vision-side budget, the result is not attributed to the ternary backbone without a matched control.
• Any fairness-audit failure marks the corresponding delta NON-CONTROLLED_EXPERIMENT and removes it from the primary causal claims entirely.

Working thesis

Extremely low-precision models should probably not be scaled by uniformly adding more low-precision parameters. Because ternarization redistributes where representational capacity actually lives, the more efficient strategy may be to spend the budget on the mechanisms that select, route, preserve and predict information — recurrent memory, selective retrieval, residual topology, prediction heads, and eventually conditional computation — with ternary weights providing the cheap substrate underneath. The shift under test is from scaling the weights to allocating the information pathways.