What Meta shipped
Meta released Muse Glimmer 30B today: an open agentic model under Apache 2.0, with a four-level reasoning dial (low → xhigh) and a launch chart that compares it against gemma-4-31B and Qwen3.6 — two model lines we have already instrumented end-to-end in this series. Meta’s recommended sampling (temperature 1.0, top-p 0.95) happens to match our frozen protocol almost exactly.
Nothing could even run it yet
Here’s what day zero actually looked like. The released serving stack didn’t know the architecture — the checkpoint literally pins an unreleased library build. The high-throughput engines produced garbage. There were no serving recipes, no tuned prompts, no folklore to lean on. So we went underneath: loaded the reference implementation directly, put our standard OpenAI-compatible interface in front of it, and ran the exact same battery we run on everything — same sampler, same grader, same selection rule, same held-out verdicts. Slower and plainer than a production serve, and that’s the point: when a model is too new for the ecosystem, the verification machinery still works. By evening, a checkpoint that nothing could load at breakfast had fully graded, execution-verified results.
First numbers — most conservative setting
First numbers, generated at the model’s lowest reasoning setting (its fastest mode — the closest available analogue to the thinking-off parity arm used throughout this series), k=8 samples, seed 0, zero truncated samples on either suite:
| suite | Muse Glimmer 30B alone (avg pass@1) | + GoM verified selection@8 | oracle ceiling | fixed / broken |
|---|---|---|---|---|
| HumanEval+ (164) | 85.7% | 89.0% (+3.3) | 90.9% | 2 / 0 |
| MBPP+ (378) | 71.7% | 74.3% (+2.6) | 79.6% | 13 / 0 |
The row that matters most is the last column. Across all 542 problems, selection fixed 15 tasks and broke zero — pairwise against the model’s own single shot, on a checkpoint that was hours old. That is the same structural guarantee we measured on NVIDIA’s Nemotron line and on Liquid AI’s LFM2.5: selection screens candidates on visible tests and is graded on held-out ones, so where an exact verifier exists, it can win but not lose. Three model families, three release eras, one mechanism.
Overnight update — the dial vs the verifier
By morning the campaign had grown to ten graded cells: two seeds at each of the conservative, medium and default (high) reasoning settings, all k=8, every cell ≤0.26% truncation. Two findings stand out.
The reasoning dial is flat on these suites. From low to high, the model’s own accuracy moves by less than a point (85.0–85.9 on HumanEval+, 71.4–71.7 on MBPP+) while spending roughly a third more tokens per task. Verified selection lifts every cell — +2.7 to +6.1 points, ten of ten — and across all ~5,400 paired instances it fixed 98 tasks and broke zero. The dial buys tokens; the verifier buys accuracy. (Scope stated plainly: these are execution-verified code suites; the reasoning dial may well earn its keep on the long-horizon agentic tasks Muse Glimmer was built for.)
| reasoning setting | HumanEval+ alone | + GoM selection | MBPP+ alone | + GoM selection |
|---|---|---|---|---|
| low (seed 0) | 85.7% | 89.0% (+3.3) | 71.7% | 74.3% (+2.7) |
| low (seed 1) | 85.6% | 89.0% (+3.4) | 71.7% | 77.8% (+6.1) |
| medium | 85.9% | 89.6% (+3.7) | 71.7% | 75.9% (+4.2) |
| high (default, seed 0) | 85.0% | 88.4% (+3.4) | 71.4% | 74.6% (+3.2) |
| high (default, seed 1) | 85.2% | 89.6% (+4.4) | 71.8% | 76.2% (+4.4) |
Replication note: the conservative arm’s second seed reproduced verified selection on HumanEval+ to the tenth (89.0 / 89.0) and came in higher on MBPP+ (74.3 / 77.8 — the seed spread is real and stated; pooled lift ≈ +4.4). The default-strength arm is likewise double-seeded (88.4 / 89.6 and 74.6 / 76.2). Across all ten cells the direction never flips: selection gains in every cell, at every setting, on both seeds — 98 tasks fixed, none broken.
The same battery, on a laptop
We ran the identical harness against the same model quantised to 8-bit on an Apple-silicon laptop — 30 billion parameters, no datacenter, through a different inference engine (llama.cpp) than the pod uses. On the 80 problems that completed locally:
| HumanEval+ (matched 80-problem subset) | model alone | + GoM verified selection |
|---|---|---|
| B300 datacenter GPU | 93.0% | 95.0% |
| Apple-silicon laptop (8-bit, llama.cpp) | 96.1% | 97.5% |
On the laptop, selection reached the oracle ceiling — of the candidates generated, it picked a correct one every time one existed — and again broke nothing. Two of the 82 problems produced no samples locally (an infrastructure failure on our side, not a model failure) and are excluded from both rows; the 80-problem subset is easier than the full suite, so these two rows compare only with each other, never with the full-164 numbers above.
What we are NOT claiming
No comparison claims yet. At every reasoning setting we measured, Muse Glimmer + GoM does not clear the gemma-4 single-shot scores on these suites, and does not clear the smaller LFM2.5 + GoM result on MBPP+ from the previous note. If the default-strength arm changes that picture, the numbers will say so; if it doesn’t, they’ll say that too. We are also not claiming the lift is large — +3.3 and +2.6 are real but modest, measured with zero truncation and confidence intervals in the ledger. The claim tonight is narrower and, we think, more valuable: the machinery transfers to a day-zero checkpoint unchanged, and its safety property holds there.
Why this matters
New models now land weekly, and every serious operator faces the same two questions within hours: is this one better for our workload — and what does it break? Most teams wait weeks for the ecosystem to settle before they can answer. We answered same-day: graded, execution-verified, regression-audited numbers before midnight, on serving infrastructure that did not exist for this model when the announcement went up. And the pattern that emerged is the one we keep measuring everywhere: put a real verifier next to a frozen model and you get gains for free — on a checkpoint that is hours old, exactly as on a two-year-old flagship. That turnaround, not any single score, is the capability.
Protocol: k=8, temperature 1.0, top-p 0.95, seed 0; reasoning_strength=low with the final-answer channel forced (the model has no true thinking-off; stated); 12,288-token budget, 0.00% samples at cap on both suites; evalplus HumanEval+ v0.1.10 / MBPP+ v0.2.0, plus-tier verdicts; selection screens on base-tier tests, verdicts always on held-out plus tests. Serving: reference transformers implementation (development build, as the checkpoint requires) behind an OpenAI-compatible shim on B300 hardware; weights via the Unsloth repack of Meta’s release (their pinned build identified the requirement); the standard high-throughput stack lacked native support at measurement time. Muse Glimmer 30B under Apache 2.0. Full tables, confidence intervals, and receipts in the research ledger.