Heterogeneous inference,
measured rather than assumed.
Weavatrix Hetero is a Rust research runtime and evidence program for deciding where each inference phase belongs: CPU, Metal GPU, or a Core AI graph that may mix GPU and Apple Neural Engine work. A route is admitted only for the exact model pack, shape, precision, hardware, OS, and evidence bundle that passed its gates.
THE MEASURED RUNTIME
One pipeline, two declared precision contracts
Core AI owns fused prefill only where a measured cost map wins. Decode remains on Metal. Unknown evidence is an error or a Metal fallback—not an optimistic guess.
Lock the model, tokenizer, prompt length, precision, and machine fingerprint.
C32 → Metal. C512 → Core AI M512. C2048 → Core AI SDPA M2048.
Decoder-ready planes avoid host K/V readback and cache materialization on the qualified path.
Dense FP16 is compiled-exact. Full HeteroPack Q8 is explicit, approximate, and experimental.
HARDWARE EVIDENCE · EXACT PATH
Latest routed control against Ollama F16
Apple M4 on AC power; Qwen3-0.6B; 16 generated tokens; six paired repetitions at each context. All 18 exact-prompt pairs rotate system order and bind source, binary, model, assets, and cost map by SHA-256. Negative percentages mean Hetero is faster.
| Context | Qualified route | Prefill | Decode / token | Complete compute | Pair verdict |
|---|---|---|---|---|---|
| C32 | Metal | +36.34% | +2.67% | +5.48% | 2 / 6 compute wins |
| C512 | Core AI M512 | −5.31% | −12.07% | −9.70% | 4 / 6 compute wins |
| C2048 | Core AI SDPA M2048 | −6.22% | −0.77% | −3.97% | 6 / 6 prefill wins |
Scope: the C2048 prefill result closed the previous +21.12% deficit, but the broader 20% product goal did not pass. This run deliberately used zero spacing between systems; its large tails remain in the evidence instead of being filtered. The exact dense path is a measured research result, not a public runtime release.
HARDWARE EVIDENCE · APPROXIMATE PATH
Q8 makes decode faster—and remains blocked
The same routed prefill was paired with all 197 HeteroPack Q8 projections. The external performance signal is strong; the portable strict quality screen is only 2/4 exact, so this mode stays explicit and off by default.
| Context | Prefill vs F16 | Decode / token vs F16 | Complete compute vs F16 | Complete compute vs Q4_K_M |
|---|---|---|---|---|
| C32 | +131.48% | −40.31% | −19.69% | +53.20% |
| C512 | −2.68% | −38.28% | −22.03% | +1.95% |
| C2048 | +2.70% | −34.16% | −5.67% | −1.04% |
Performance PASS ≠ quality PASS.
C512 cleared the 20% complete-compute target against Ollama F16 in all 6/6 repetitions. No context cleared that target against Ollama Q4_K_M. Lower weight-byte width is still the unsolved Q4 race, and full Q8 cannot promote until a widened held-out quality suite passes.
WHAT THE EXPERIMENTS ACTUALLY TAUGHT
Wins, limits, and retained failures
Negative results remain first-class evidence. They change the scheduler and stop attractive microbenchmarks from becoming unsafe defaults.
Fused prefill is real
The GPU-preferred fused graph matched 26/26 tokens in one FP16 contract. Against this project's reference Metal runtime, internal bucket tests measured 4.8× / 2.1× / 1.75× TTFT at M128 / M512 / M2048; those are not claims against the best possible Metal implementation.
The compiler chooses a mixed graph
Core AI auto placement used both GPU and ANE. It matched 23/28 prompts and 89.1% of continuation tokens, with near-tie greedy flips. It is a declared approximate mode, never an exact-path substitute.
Per-layer offload amplifies drift
Serial Metal-attention → ANE-MLP boundaries failed full-chain correctness. A whole Core AI decode step was exact but 1.6× slower than Metal and crashed under mixed placement. Prefill can route; decode stays Metal.
Unmanaged overlap breaks the SLO
Running mixed prefill beside foreground Metal decode regressed decode TPOT by +18%. Aggregate throughput rose, but request overlap is not admissible without a flow-aware scheduler.
QUARANTINED SIMULATIONS
Hypotheses that have not earned hardware status
Every simulation result carries hardwareEvidence: false and productionEligible: false. The production planner rejects it; the numbers only prioritize physical experiments.
Assumed acceptance and latency; no trained drafter or output-quality run.
91.646% of requests crossed devices—the join/residency warning behind the average.
Synthetic MLA fit; 26.546% activation RMSE, not a converted Qwen model.
A 144-node synthetic DAG; no claim about observed runtime speedup.
Private source, public limits
The repository stays private because it contains experimental runtime and asset pipelines, not because its conclusions are exempt from scrutiny. Public claims on this page name the machine, model, precision, comparison boundary, repetitions, quality class, and failed gates. Benchmark harness tests cover prompt identity, balanced schedules, lifecycle isolation, raw-stream hashing, provenance, route receipts, retry semantics, paired statistics, and fail-closed promotion. Apple hardware conclusions come only from the committed macOS evidence bundles—not from a Windows build or a synthetic simulator.