A step-by-step case study of Clef-Flash, Nimble, external memory, deterministic guardrails, and the limits of model-only decision-making.
Research completed October 22, 2026.
The short answer: how do you make an AI agent more reliable?
Build a reliable decision system, not a model-shaped replacement for every component. Represent the state explicitly, enforce deterministic constraints in code, use a model where uncertainty remains, and evaluate the complete workflow on held-out cases. Add retrieval or weight training only when a controlled comparison shows that it addresses a real failure.
That is the main engineering lesson from our Minesweeper experiments. The project started with two decision models choosing cells. It evolved through additional lives, external memory, flags, diagnostic tests, logical guards, and anti-cycle protection. The final memory study contained 2,520 games: 360 experience-collection games and 2,160 evaluation games.
The most important result was a distinction rather than a leaderboard: improving the system around a model is not the same as improving the model itself.
This was an independent engineering experiment prepared for this case study, not a DS Stream client deployment. The recommendations below are engineering interpretations, not measured business outcomes.
Why use Minesweeper to test AI decision-making?
Minesweeper makes consequential decisions observable. A player sees incomplete information, chooses an action, and receives a verifiable outcome. Some moves can be proved safe from visible clues; others remain uncertain. That separation lets us ask whether a system ignores available evidence before blaming its performance on unavoidable risk.
Our board was 8×8, with difficulty levels of 10, 12, 14, 16, 18, and 20 mines. Seeded generation made boards reproducible. A protected central opening gave paired players the same initial visible state. After different actions, their later states could diverge.
Clef-Flash and Nimble were used as decision models through Ollama's System One interface, not as conversational agents writing unrestricted moves. The current Ollama decision documentation describes binary estimates, label selection, and rubric scoring. A returned confidence value is not a guarantee of correctness.
The game is a useful diagnostic environment, not a proxy for every enterprise workflow. Our three-life variant is also not standard one-hit Minesweeper.
How the experiment evolved, step by step
1. Make the interface valid before measuring intelligence
An early design tried to select directly from all available cells. The tested choice interface allowed no more than 26 options. That made the initial formulation unsuitable for larger action sets.
We switched to binary noul questions about candidate cells, then selected the cell with the lowest model-estimated mine score. Rules were represented once in the shared input instead of being repeated in every question, after a capacity test exposed excessive input size.
These were integration corrections before the main comparison. Invalid requests and context failures are not evidence that one model is a worse player.
2. Establish baselines before adding memory
The first completed main series, v3, used 30 paired boards for the two models, plus random and rule-based reference players. Both models won 0/30 games. The rule-based solver won 11/30.
That result challenged the starting assumption. A model producing a score for every candidate was not automatically using obvious logical information well. The reference solver gave us a concrete alternative to “try a larger prompt.”
3. Add three lives, but label the task change honestly
Version 4 introduced three lives and a warm-up period. Mine hits before the player's first safe move did not consume a life; the automatic opening did not end warm-up. After warm-up, the third mine hit ended the game, and hit mines remained visible.
Both models still won 0/30 games. The reference solver won 24/30. This was not evidence of model learning: the game mechanics had changed. A comparison between v3 and v4 cannot isolate a model improvement.
4. Test experience retrieval without changing model weights
Version 5 added an isolated Mnemo memory database. The models generated experience through gameplay; an adapter converted revealed outcomes and visible-state audits into feedback cards. Retrieval combined SQLite full-text search, bge-m3 embeddings, and reciprocal rank fusion.
After 200 experience-collection games, memory was frozen. Evaluation used 600 games on new seeds, comparing no feedback, own-author examples, and shared examples. Both models had zero evaluation wins in each condition. The main shared-minus-own coverage difference was approximately +0.26 percentage points, with a 95% interval spanning zero.
This was external memory, not fine-tuning, and “own” meant filtering the same jointly collected history by author, not training a separate isolated model.
5. Add flags and lessons, then inspect why the system still fails
Version 6 introduced flag and unflag actions alongside a revised lesson adapter. The study again contained 200 experience-collection games and 600 evaluation games. It produced zero evaluation wins; one unfinished game remained in the analysis rather than being discarded.
The audit showed decisions that skipped provably safe cells and revealed provable mines. It also exposed an important distinction: a flag was a hypothesis, not a verified mine. Treating flags as ground truth would have turned the system's guesses into false premises.
Flags and the adapter changed together, so v6 versus v5 was not a clean estimate of either individual effect.
6. Diagnose representation, action selection, and protection separately
Version 7 broke the problem into stages. We tested known positions, compared a grid with explicit clue constraints, compared binary scoring with tournament-style action choice, and then tested a logical guard on fresh boards.
The representation result was mixed across models. On selected safe-available positions, Clef-Flash improved from 7 to 15 safe reveals out of 30, while Nimble moved from 9 to 5. The pooled difference was uncertain. Tournament-style choice also underperformed binary scoring on the selected diagnostic positions. These were diagnostic selections, not fresh generalization claims.
The fresh guard pilot was more decisive: baseline won 0/60 games; the guarded system won 51/60. Paired coverage increased by 52.86 percentage points, with a reported 95% bootstrap interval of 47.11 to 58.56 points.
The guard prioritized provably safe reveals and blocked reveals of provable mines, using visible information. That improvement belongs to the hybrid system, including its explicit rules. It is not evidence that the neural weights learned Minesweeper.
The subsequent six-history memory study did not demonstrate the required practical advantage of shared over own examples. Its logs verified exposure to retrieved cards, but did not contain everything needed to replay the complete semantic ranking. That limitation shaped the next version.
7. Address action loops before expanding the memory claim
The v8 pilot tested an anti-cycle wrapper with the same rules and no learned examples in either arm. Across 120 games, full-game flag cycles fell from 67 to 3, while wins moved from 47/60 to 48/60.
The wrapper remembered actions within a visible-board epoch. Changes to flags or remaining lives alone did not reset that epoch. It reduced repetitive behavior in the pilot, but did not promise that every full-game flag cycle would disappear after later board progress. The wrapper was selected and frozen before the final memory study.
Does agent memory improve decisions? The final controlled test
In the final v8 B experiment, additional retrieved examples did not demonstrate the preregistered practical benefit over the simpler controls. Both adjusted confidence intervals included zero. This is a result about the tested models, cards, retrieval policy, boards, and wrapper. It is not proof that memory is useless in general.
Each of six fresh histories contained 30 paired experience-collection rounds: 60 games per history, 360 in total. Evaluation then used 60 new seeds per history, both models, and three conditions: 2,160 games in total.
| Evaluation condition | What the model received |
|---|---|
| Rules-only | The same four general rules, with no retrieved examples |
| Own | The same rules plus examples filtered to the model's author |
| Shared | The same rules plus examples from both authors |
Every condition used the same representation, decision interface, guard, and anti-cycle wrapper. Within each history, own and shared read the same frozen database. Paired training games saw the same experience cutoff, and feedback was committed atomically only after both games finished. Evaluation made no database writes.
The context budget was at most six cards: four rules and two examples, with a 2,400-character text limit. These controls made the question narrower and more useful: what did extra access to examples add on top of the established system?
Results: small estimated effects, no demonstrated practical benefit
| Primary contrast | Coverage difference | Adjusted 97.5% confidence interval |
|---|---|---|
| Own examples minus rules-only | +0.289 percentage points | −1.663 to +2.093 percentage points |
| Shared examples minus own examples | −0.390 percentage points | −1.354 to +0.459 percentage points |
The preregistered practical-benefit rule required a point estimate of at least five percentage points and an adjusted lower confidence bound above zero. Neither contrast passed.
Coverage measured the fraction of initially unrevealed safe cells eventually revealed, excluding the automatic opening. Analysis used a paired, nested bootstrap with 5,000 resamples, equal history and seed weights, fixed models, and Bonferroni adjustment for two primary contrasts. Six histories limit precision and interval calibration; nominal family-wise coverage is not an exact guarantee.
For context, the rules-only hybrid won 295/360 Clef-Flash games and 286/360 Nimble games. These are descriptive system-level counts, not isolated measures of neural reasoning. Nor do early-versus-late win counts form a controlled learning curve: boards, mechanics, and software assistance changed between versions.
Watch the two models: first three-life games versus the final system
The following videos show real logged actions. Left: the earliest v4 three-life series. Right: final v8 shared-condition games. Both sides use 8×8 boards and three lives, progressing only upward through 10, 12, 14, 16, 18, and 20 mines.
Clef-Flash
Open the Clef-Flash video (MP4)
Nimble
At each difficulty, selection takes the first eligible v4 game and the last eligible v8 game, not the best-looking outcome. Seeds differ between sides. v8 has flags, rules, a guard, anti-cycle protection, and shared examples; v4 does not. These are illustrations of changed system behavior, not causal evidence of learning or memory benefit.
Playback uses 1.4 seconds per decision, not real inference time. The original overlays are in Polish; the English captions explain the comparison. Full selection and source hashes are recorded in the experiment's video manifest.
What should run in code, and what should run in a model?
Use code for authoritative state transitions, arithmetic, access checks, and constraints that can be specified and tested. Use a model for the residual interpretation or ranking problem only when it improves the measured outcome. A model should not be asked to rediscover a rule that the application can enforce reliably.
| Responsibility | Recommended owner | Why |
|---|---|---|
| State, legal actions, counters, transactions | Deterministic application code | A consistent source of truth |
| Provable domain constraints | Tested rules or a solver | Explicit, inspectable deductions |
| Ambiguous classification or candidate ranking | Model, if validated | A bounded role under uncertainty |
| Historical context | Retrieval, if it adds value | Additional evidence, not authority |
| Permission to execute | Policy and validation layer | A proposal is not authorization |
| Failure recovery and repeated actions | Controller with limits | Stops loops and silent retries |
| Quality claims | Held-out evaluation | Separates a demonstration from evidence |
This is a proposed design pattern, not a claim that our experiment proved the best allocation for every business process. For a fully solvable symbolic task, the best choice may be a solver with no neural model at all. We did not establish that the final hybrid beats a strong exact solver.
For example, a model may interpret an unusual invoice description, while code checks totals, duplicate identifiers, and approval thresholds. In a data pipeline, a model might classify an ambiguous incident, while schemas and permissions remain deterministic. These are applications of the principle, not deployments measured by this study.
How should you train models for better decisions?
First separate three interventions: better inputs, better external context, and weight updates. They solve different problems. Our study tested the first two and controller changes; it did not perform supervised fine-tuning, preference optimization, or reinforcement learning on either decision model.
Fix the decision environment first
Before planning training, define the observable state, valid actions, objective, and unacceptable errors. Establish a non-model baseline. Inspect whether failures come from missing information, bad representation, illegal actions, repeated actions, or genuine ambiguity. No amount of parameter training repairs an inconsistent state machine by itself.
Build training data from verified decisions, not just successful games
For a future weight-training study, a useful record would contain the pre-action observation, allowed candidates, chosen action, observed outcome, and an independently checked justification where available. Keep hidden simulator truth out of deployed inputs. Do not label a lucky guess as a logical deduction, and do not treat every losing-game action as wrong.
Split by complete boards or trajectories and independent experience histories, not randomly by adjacent moves. Otherwise, near-duplicate states can leak across training and evaluation. Preserve separate development and final test sets; freeze the final evaluation before tuning.
Choose a training method that matches the error
- Supervised fine-tuning: a candidate when the model repeatedly fails to reproduce verified state-to-action behavior. It requires an appropriate training interface and access to trainable weights.
- Preference optimization: a candidate when the same observation supports meaningfully better and worse action proposals. Direct Preference Optimization is a published approach to fine-tuning from preference comparisons; its success on this task would need a new experiment.
- Reinforcement learning: a candidate when long-term consequences matter and the simulator provides a trustworthy reward. A proposed reward must distinguish progress from flag churn or warm-up exploitation. Nothing in this study shows that such training would outperform a solver.
These are possible next studies, not completed work or a ready-to-run recipe for these particular model artifacts. Extra examples retrieved at inference time do not update parameters. The original RAG research is relevant background on combining parametric models with external information, not validation of our specific memory adapter.
Evaluate the intervention against its simpler alternative
Keep wrappers, rules, action budgets, and test boards matched. Compare the proposed trained model with the original model under the same controller, and compare the controller with a solver or non-model baseline. Measure forbidden actions, completed tasks, useful progress, latency, and cost, not only average score.
Our study did not establish production ROI, standalone throughput, or an optimal fine-tuning method. Training is justified by a measured residual problem, not by the fact that a model is available.
How do you know the result is trustworthy?
The final audit checked 4,892 transferred files against source hashes and 37 frozen dependencies. It replayed all 2,520 games and 71,501 decisions, including engine state, submitted payloads, recorded responses, guard behavior, and random choices. It also replayed 81,290 retrieval searches and reconstructed six databases and 180 atomic feedback commits.
One practical issue mattered: Windows and the original Linux numerical environment produced tiny cosine differences. We preserved the mismatch evidence and performed exact retrieval replay in the original environment instead of relaxing the acceptance rule.
The replay reused recorded query embeddings; it did not independently rerun the embedding model. Runtime snapshots supported residency and digest checks at recorded moments, not continuous physical GPU isolation. Our logic auditor was incomplete: “not proved safe” did not mean “must guess.”
These boundaries make the result interpretable. They do not turn it into a universal guarantee.
FAQ: reliable AI agents, memory, and training
Did the models learn to play Minesweeper?
Their weights did not change. We collected experience, retrieved feedback, changed representation, and improved the controller. The resulting system became more capable under later protocols, but that is not proof of parameter learning or a controlled before-and-after model improvement.
Did shared memory improve performance?
The final v8 B study did not demonstrate its preregistered practical advantage over own-author examples. The estimated coverage difference was −0.390 percentage points, with an adjusted 97.5% interval from −1.354 to +0.459. That does not prove equivalence or rule out other memory designs.
Are guardrails just better prompting?
Not here. The logical guard was executable selection logic based on visible constraints. It could restrict or redirect actions regardless of what a prompt requested. The four general rules in context were a separate component.
Should every AI agent use retrieval?
No. Retrieval should address a demonstrated information gap and pass a comparison with a simpler baseline. A retrieved example can be relevant without adding useful information to a decision already constrained by explicit rules.
What is the first step before fine-tuning?
Create a reproducible evaluation and a failure taxonomy. If the dominant errors are arithmetic, permissions, or known constraints, repair those deterministically before spending effort on weight training. Then test whether any remaining model limitation is learnable from verified data.
The engineering takeaway
The useful question is not “How can we make a model do everything?” It is “Which responsibilities should this model actually own?”
For this experiment, stronger system design produced the clearest improvement, while extra memory did not meet the final benefit criterion. Reliable AI engineering means retaining that distinction, even when a more dramatic learning story would be easier to market.
For teams evaluating a similar architecture, the practical next step is a bounded decision audit: map deterministic responsibilities, isolate the model's role, establish a baseline, and agree on the acceptance threshold before adding complexity. Explore DS Stream's AI and data engineering expertise.
Sources
