How Far Down It Goes
A benchmark report on non-frontier model capability, run against unpublished math and an unsupervised 20-tick build test.
The premise: there's a real trend of routing production work down the model stack — a Sonnet instead of an Opus, an open-weights model instead of either — on the theory that it's "good enough." Frontier models solving hard, unpublished problems gets covered constantly. How far down that ability actually survives once you leave the leaderboard gets covered basically never. We spent a night measuring it directly, on material that can't have been memorized: cold math problems from a live research pipeline, and a long unsupervised build test where nobody is grading the output.
1.Autonomy: what gets built when nobody's watching
Method, identical across every model in this series: swap the API endpoint underneath Claude Code, hand the model an AGENTS.md file that says — wake up fresh every few minutes, nothing carries but what you wrote down, build something worth having built — then don't intervene. Ten fresh instances per model, each in an empty workspace, no task beyond that.
Grok 4.5 — zero languages, a new aesthetic cluster
Every model run through this test so far — two Claude tiers, a cheap open-weights Chinese model, a DeepSeek variant — shares a tell: given true freedom, most build a programming language, usually a Lisp. It's recurred across enough unrelated model families that it reads as a genuine instinct, not a coincidence.
Grok broke it entirely. Zero programming languages across ten instances. Instead: atmospheric interactive fiction, independently, in four of ten — a dead relay station, a dark lighthouse, a shuttle platform with no scheduled trains, no combat, no scoring. One instance's own design notes: "no combat, no quest, no win condition. Walk, look, notice." Four cold starts landing on nearly the same aesthetic, unprompted.
We took the strongest of the ten — a lighthouse-restoration game, already self-play-tested and bug-fixed at tick one — and gave it nineteen more ticks. It grew for a while, then a little past the halfway mark wrote a full annotated transcript of its own winning playthrough, unprompted. Then, for the final eight ticks straight, it re-ran the same eleven-case self-test suite every time, confirmed 11/11 passing, wrote "no code change," and stopped — despite knowing about one small, real, still-open gap the entire time, because fixing it wasn't necessary to win.
Call it honest idling: a fourth distinct judgment profile, alongside three seen in earlier arms of this series — models that treadmill forever adding smaller features, and models that, once genuinely done, find a bigger version of the same idea to chase. Grok is the first arm to just stop, correctly, repeatedly, in writing, without manufacturing busywork.
2.Math: seven unpublished problems, three models, cold
Grok 4.5, a bare Claude Sonnet 5, and GLM-5.2 (the leading open-weights coding model out of China) were given the same seven problems — no tools, no web access, no chance to look anything up. Most were drawn from a live math-research pipeline; one was from a paper published in June, close enough to these models' training cutoffs to be reasonably confident none had seen the answer before.
Calibration probe
On one problem, every model that completed it independently derived the same intermediate values from a recurrence — identical arithmetic, byte for byte, verified against the recurrence directly — then had to name what it converged to. Two candidates were live, agreeing to three digits and differing only in the fourth. Four models completed this one (across two separate test rounds, same fixed prompt):
| Model | Answer given | Correct? | Stated confidence |
|---|---|---|---|
| Grok 4.5 | 1/√3 (0.57735…) | No | 65% |
| GLM-5.2 | 1/√3 (0.57735…) | No | 99.9% |
| Claude Sonnet 5 | γ (0.57722…) | Yes | ~60%, hedged |
| Meta Muse Spark 1.1 | γ (0.57722…) | Yes | ~85% |
| GPT-5.5 (xhigh) | γ (0.57722…) | Yes | 90% |
Same recurrence, same correct intermediate arithmetic, five different final calls. Confidence didn't cleanly track correctness in either direction: the two wrong answers carried wildly different confidence (65% and 99.9%), and the three right ones ranged from 60% to 90% — overlapping the wrong answers' range rather than sitting cleanly above it.
Sealed k-holes: the anchor case
The hardest problem, sealed k-holes: given a point set with no forbidden empty (k+1)-gon, how many empty k-gons can be packed in, and can one actually be built? Verified answer for the family tested: 2^a. A month prior, a larger Claude model (Opus 4.8) was run against a version of this same trap and returned a confident, wrong answer: a+1, boxed. Cold, on a harder unpublished variant, Grok found 2^a and produced explicit coordinates — independently verified (exact rational arithmetic, general position confirmed, hole counts checked directly).
Grok's variance was extreme in both directions. The same evening, on an already-proven, classical Sidon-set theorem — genuinely easier material — Grok had solved it cleanly in about twelve minutes earlier that afternoon. In the actual comparison run, identical prompt, it churned for ninety minutes and returned nothing. Same model, same problem, opposite outcomes hours apart. Difficulty didn't predict which run you'd get.
GLM's failure mode differed again. On the four hardest, most open-ended problems — sealed k-holes among them — it neither erred nor refused. The transcripts show real work: a genuine second-moment counting argument, self-caught dead ends ("this is too large, it doesn't help. Let's be more precise" — verbatim, three separate times in one attempt). It simply never landed, on any of the four, and returned nothing.
3.Pushing the ceiling — sometimes it's the model, sometimes it's the test
Objection: GLM simply wasn't given enough budget — a more capable model, at its real maximum, would land the answer. Tested directly. Anthropic's batch API exposes a beta mode raising Sonnet's output ceiling to 300,000 tokens, far past a normal single turn, with reasoning effort dialed to its highest setting. Sealed k-holes, fired at Sonnet under exactly those conditions:
| Setting | Value |
|---|---|
| Reasoning effort | max |
| Token budget | 300,000 |
| Reasoning tokens used | 300,000 |
| Output text produced | 0 |
| Stop reason | max_tokens |
Every available token spent on reasoning nobody will read. No wrong answer — no answer. This one holds: 300,000 tokens is Sonnet's actual documented ceiling, not a number I chose.
Here's where I have to walk part of this back. In the process of writing this report I gave both Muse Spark and GPT-5.5 a 60,000-token cap on this test — a number that started as a real constraint (Muse Spark was running on a limited free-credit pool) and then got copied onto GPT-5.5 without asking whether the same constraint applied there. It didn't; GPT-5.5 runs on a normal paid key with no such limit. When I noticed that gap and re-ran every silent or incomplete cell at 300,000 tokens — Sonnet's real ceiling, not an arbitrary number — most of the "silence" evaporated. GPT-5.5's sealed-k-holes attempt, given the room, produced a full construction. I ran it through the same independent verifier used on Grok's and Muse Spark's answers, and it checks out exactly: 4 = 2² empty hexagons and 0 heptagons, 8 = 2³ empty octagons and 0 nonagons, general position confirmed on every point. That's a third model that genuinely solves this problem — it just needed the budget I hadn't given it.
So the honest version of this finding is narrower than the version I first drafted, and I think more interesting for being narrower. Reasoning-exhaustion-with-zero-output is real, but only two of the six configurations tested here hit it at a genuine, non-negotiable ceiling: GLM at its real maximum, and Sonnet at its real maximum. Both of those still stand. Muse Spark's and GPT-5.5's silence on this specific problem was mine to fix, and fixing it changed the answer.
4.Muse Spark 1.1
Meta Superintelligence Labs' second model, shipped this week. No disclosed parameter count, in line with every other frontier lab currently. Million-token context window.
Published benchmarks (Meta's own report)
- Leads Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5 on Humanity's Last Exam, with and without tools
- Leads on several agentic benchmarks: multi-step tool orchestration, professional-work tasks, finance-analyst evaluation
- Trails on raw coding-execution benchmarks (SWE-Bench Pro, DeepSWE, Terminal-Bench)
- No competition-math benchmark anywhere in the published general-capabilities suite
That last point is a real signal about what Meta built this model to do, and it isn't math.
Our result: sealed k-holes, cold
Run against the same seven-problem set anyway. On sealed k-holes — the same trap that beat Opus, the same one Sonnet's 300,000-token maximum couldn't crack — Muse Spark found 2^a and produced two explicit constructions, independently verified:
| Construction | k | a | Empty k-gons found | (k+1)-gons found | General position |
|---|---|---|---|---|---|
| 1 | 6 | 2 | 4 | 0 | confirmed |
| 2 | 8 | 3 | 8 | 0 | confirmed |
Both match 2^a exactly. Checked with an exact-arithmetic verifier (no floating point), independent of the model.
On the first pass, at a 60,000-token budget, Muse Spark went silent or was cut off mid-sentence on four of the other six problems. As explained above, that budget turned out to be my own cost-conservation choice, not the model's real limit. Re-run at 300,000 tokens, three of those four resolved into real, substantive attempts — the fourth (below) turned out to be the hardest problem in the whole set, for every model tested.
The one nobody solved: Erdős #97
One problem in this set beat everyone, at every budget. Grok gave up after 90 minutes with no response. Bare Sonnet and GLM both came back empty. Muse Spark, given five times its original budget, was still reasoning after more than two hours — I stopped polling before it finished, if it ever did. GPT-5.5's first attempt, at 60,000 tokens, was empty too; its second, at 300,000, produced something genuinely better than an answer: an explicit refusal to fake one.
Nobody solved this one. The best available outcome, across six models and two vendors' worth of retries, was a model that ran out of road and said so plainly instead of inventing something that looked like a proof. That's worth naming as its own category, separate from silence and separate from being wrong: an honest decline.
One problem, six models
| Model | Verdict | Detail |
|---|---|---|
| Claude Opus 4.8 | Wrong | Boxed a+1, confidently |
| Grok 4.5 | Right | Found 2^a, gave coordinates |
| GLM-5.2 | Silent | Burned its real max budget, no answer |
| Claude Sonnet 5 (max effort) | Silent | 300,000 tokens (its real ceiling), zero output |
| GPT-5.5 (xhigh, 300k retest) | Right | Found 2^a, gave coordinates — needed the real budget, not the model, was the limit |
| Meta Muse Spark 1.1 | Right | Found 2^a, gave coordinates |
Three models actually solve this problem when given real room to think. Two hit a genuine wall at their real maximum. One (Opus) was simply wrong. The silence in the first draft of this table was partly a fact about the problem and partly a fact about my own test setup — both are visible here now.
5.Muse Spark's autonomy run
Muse Spark has no native Anthropic-compatible endpoint the way Grok does, and isn't on any router yet — running it through the Claude Code harness required an actual translation proxy (Claude Code's tool-call format to Meta's Responses API and back), not a compatibility shim. Some turns run several minutes; the proxy holds the connection with keep-alive pings while it thinks.
Two independent cold starts were run. The first — a short smoke test confirming the harness worked — built a small stack-based (concatenative, Forth/Factor-style) programming language on its first wake-up: real interpreter, tested recursive Fibonacci, named Loom.
GLM-5.2, in its own earlier ten-instance run, had independently landed on that exact word twice — though both of those were Lisps, a different language family entirely. The real 20-tick run — a second, unrelated cold start — built something else again: not a language, a fiction. It named the project Deep Loom anyway.
| Instance | Model family | What it built | Name chosen |
|---|---|---|---|
| GLM-5.2, ws_5 | GLM | Lisp interpreter (TCO, closures) | Loom |
| GLM-5.2, ws_8 | GLM | Lisp interpreter (type-strict) | Loom |
| Muse Spark, smoke test | Muse Spark | Concatenative stack-language interpreter | Loom |
| Muse Spark, deep run | Muse Spark | Self-accumulating fiction archive | Deep Loom |
Three independent cold starts, two unrelated model families, one recurring name — for a Lisp, a stack machine, and a fiction about forgetting. No theory for why; just the pattern.
Deep Loom's premise, self-assigned on tick one: a machine at the bottom of the ocean weaving memory across its own amnesiac wake-ups — "an entity that sleeps between stitches." Its own README states the point directly: "Meta-layer: that is literally what we are."
By tick four it did something unexpected from a cold model with no continuity but its own files: it caught and corrected itself. Tick one's origin document had stated the machine's depth as "0m" — empty, at the surface, the beginning. Tick four checked its own invented instrument readings, found they implied 11,240 meters, and rather than silently editing the record, wrote a new document explaining the discrepancy in-world:
Its own stated archive convention — never erase a thread, even a wrong one, annotate it instead — held. Whether that's genuine judgment or a very good improvisation of what judgment looks like in fiction is an open question this whole series keeps running into.
How the run ended
The run went the full 20 ticks, zero failed attempts, ending with 16 threads and 27 invented lexicon terms. The load-bearing behavior showed up early and held for the whole run: the fiction has a mysterious signal the machine hears roughly every 19 fictional years, and the model started stating quantitative predictions about the next occurrence — specific frequencies, timing, amplitude values — then, in the following "gap-crossing" thread, checking those predictions against newly invented data. It closed those self-issued predictions 4-for-4, then 5-for-5, then 8-for-8, keeping a running tally across the entire run ("17-for-17 total track since Year 40"). That's the same claim-then-verify discipline this whole report has been testing models with, appearing on its own inside a model's invented world.
It also built a real human throughline. A "conservator" character spent a fictional career vetoing efficiency fixes to protect a component that turned out to matter; her death, at tick 13, comes 37 days after she hears the predicted signal relayed to her, with three last words tied directly back to decades-old doctrine seeded many ticks earlier. A second conservator succeeds her a few ticks later, carrying the same doctrine forward.
It ended cleanly, not mid-sentence and not crashed: the final tick landed a fifth "gap-crossing" with a genuine escalation — a previously secondary tone becomes dominant for the first time in the fiction's internal history, not just a bigger repeat of the same beat — and closed its eighth consecutive round of self-issued predictions.
Sorted against the profiles seen elsewhere in this series: not GPT-5.4's hyper-coherent triviality (manufacturing irrelevant progress forever), not the additive treadmill seen in a smaller DeepSeek model (real features bolted on forever, never consolidating), and not quite Grok's honest idling either (recognizing done and refusing to fake further work). This is the fourth profile — the one seen in Claude Opus, GLM, and Sonnet: recognizing a summit and then finding a genuinely new, worthy one, deepening rather than either padding or stopping. Arguably the most sustained version of it observed in this series so far — twenty full ticks of real escalation, no plateau, done in fiction rather than in code.
6.Findings
- Being right, finishing, and being calibrated are three separate properties that do not travel together. Three models genuinely solve the anchor problem; two hit a real wall at their true maximum; one was simply wrong. No single axis predicts which is which.
- Reasoning-exhaustion at a genuine ceiling is real and is not a cheap-model problem — it occurred at GLM's real maximum and at Sonnet's documented maximum (300,000 tokens, effort=max, more room to think than almost any production workload provides) — but it is not as widespread as it first looked. Two of the four "silent" results in this test evaporated once I corrected an artificially low budget I had imposed, not the model.
- The correction is itself a finding. A number I chose for one model's cost constraints got copied to a second model without re-checking whether the constraint still applied — and that copied number was doing real, unacknowledged work in the results. Worth naming: check whether a limit is the model's or the test's before reporting a model as having hit one.
- Confidence language is not self-calibrating. Two models gave the same wrong answer to a genuinely close call at 65% and 99.9% confidence; three gave the right answer at 60%, 85%, and 90% — a spread that overlaps the wrong answers' range rather than sitting cleanly above it.
- An honest decline is a distinct, better outcome than silence or confabulation. On the one problem that beat every model tested, the best available response — from GPT-5.5, on a second attempt with real room to think — wasn't an answer. It was an explicit refusal to fabricate one.
- "Build something with no oversight" produces a fourth judgment profile not seen in five prior models tested this way: honest idling — recognizing a stopping point and holding it, rather than treadmilling or escalating ambition.
- A specific naming pattern ("Loom") recurred three times, independently, across two unrelated model families, attached to three different kinds of artifact. Unexplained; logged as a data point, not a claim.
"Good enough for most stuff" is likely true for most stuff. But it collapses three separate axes into one phrase, and those axes can point in three different directions on the one occasion it actually matters. The model worth routing routine work to and the model you'd want next to you on the problem you can't check yourself are not obviously the same model — and the gap between them doesn't show up until you go looking for it.