Key Takeaways
- On one unpublished NP-hard optimization benchmark, the source says Fable 5 produced the best overall result.
- The write-up argues that /goal is not a general “try harder” switch; it can help or hurt depending on the search path it encourages.
- Across the reported matched runs, /goal won more individual trials but did not improve the average outcome.
What happened
The source describes a benchmark run on an unpublished operations research problem originally given to students at a hackathon. The task is a fiber-network design problem involving directed distance matrices for Grenoble, Nice, and Paris, with the solver trying to connect distribution points and terminals while meeting structural constraints and minimizing total cable length.
To test agent behavior, the author ran Claude Fable 5 and GPT-5.6 Sol on the same problem, both with and without their native /goal mode. The write-up says Fable 5 was the strongest performer overall and that its output was notably consistent. In the reported comparison, Fable’s plain mode beat Sol’s plain mode by 1,875 points on average, and Fable’s /goal mode beat Sol’s /goal mode by 1,984 points.
The author also reports that /goal did not behave like a simple “try harder” feature. Instead, it changed the control loop and the search path. In the described trials, /goal won four of six head-to-head comparisons, but the mean scores worsened for both models because the feature sometimes helped and sometimes extended a bad approach for longer.
The article goes into some detail about why this benchmark is unusually difficult. It points out that the search space is enormous, even under simplified assumptions, and argues that this makes optimization behavior highly sensitive to solver choices. The author says that a valid network can involve loops, short branches, and constraints that make apparently small changes in structure affect cost.
A separate section explains that the same /goal label hides different implementations in different tools. For Claude Code, the source says /goal is implemented as a session-scoped Stop hook using a separate evaluator model that can approve or reject progress based only on the transcript. For Codex CLI, the source says goal state is persisted in thread state, exposed through tools, and resumed when the thread goes idle. The author’s point is that the two systems are not equivalent even though they share similar language.
Why it matters
The broader takeaway is not that one model is universally better than another. It is that a persistence or goal-tracking feature can improve some individual runs while still making average performance worse on hard optimization problems.
That distinction matters because many agent workflows assume more persistence is always beneficial. In coding or routine task completion, another turn often helps finish a migration, fix a test, or refine a plan. In an optimization setting, though, extra iterations can amplify a good search direction or keep a bad one alive longer. The source argues that this is exactly what happened in the benchmark: /goal sometimes found a better basin, but it also sometimes gave a weak idea more time to mature.
The article is also a reminder that benchmark results depend on more than the model name. The author says the benchmark measured the complete system, including the CLI, prompt, subscription service, and harness. The source notes limitations as well: the test is one unpublished NP-hard task, only Fable and Sol have three clean matched pairs, and the runs may have been affected by service drift and environment details.
For readers tracking agent tooling, the most interesting claim is the narrower one: goal-setting mechanisms are not interchangeable, and they can alter search behavior in ways that are not obvious from individual wins and losses alone.
What to watch
The source suggests a few questions worth following in future testing:
- Whether /goal-like features consistently help on tasks where progress is hard to judge in the short term.
- Whether different agent frameworks expose goal persistence as a separate evaluator, persisted state, or something else entirely.
- Whether broader benchmark suites confirm the pattern seen here: more wins at the trial level, but weaker average performance because of larger negative outliers.
The author also says the benchmark code, prompts, result tables, and analysis scripts are available in CLIArena, which suggests this test may be reproducible or extended by others. If more comparisons appear, the most useful next step will be to separate the effect of the model from the effect of the control loop and runtime harness.
