Keeping AI Experiments Reproducible

Track the run, not just the result

When an experiment with AI tools goes well, the instinct is to save the output: the generated code, the summary, the analysis. That is useful, but it is not the part that matters most for reproducibility. What matters is the run itself, the exact configuration that produced that output. A result without its run is a photograph without a location. You know something happened. You have no way back to where.

Treat the run as the unit you are keeping, not the answer. That means capturing the prompt exactly as sent, the system instructions if any were set separately, the model and the version of it you were using, and the date. None of this needs a database or special software to start. A plain text file with consistent structure, updated as you go, beats a perfect memory every time, because perfect memory does not survive much past a week.

Prompts and parameters are part of the method

In traditional experimental work, nobody would report a result without describing the method that produced it: the reagents, the equipment settings, the conditions in the room. AI-assisted work has an equivalent, and it gets skipped constantly because it feels like an implementation detail rather than part of the finding. The prompt is not an implementation detail. Neither is the temperature setting, the context window contents, or whether retrieval was involved and what it returned.

This matters because small changes to any of these can move an output more than people expect. Two researchers who believe they ran the same experiment, because they used the same rough question, can get meaningfully different answers because one of them had a longer conversation history in context or a different system prompt underneath. Until the full configuration is written down, "the same experiment" is really just a similar-sounding one.

Why a good demo still isn't evidence

A demo is optimized for one thing: looking good in the moment it is shown. That is a different goal from being representative. The person giving the demo has, consciously or not, selected the prompt, the framing, and often the specific run that worked best. There is nothing wrong with that when the goal is to communicate a possibility. It becomes a problem when the demo gets treated as proof that a technique works in general.

The honest question after any impressive demo is not "how did they do that," it is "what happens if I try it right now, cold, with my own data and my own phrasing." Usually the answer is instructive: sometimes the technique holds up, and sometimes the gap between the demo and your own attempt tells you exactly what was doing the real work, whether that is a specific dataset, a narrow task, or a prompt tuned over many hidden attempts.

Give seeds and model state the respect they deserve

Randomness is not a footnote in AI-assisted work, it is often the whole story of why a result will not repeat. Where a system exposes a random seed or similar setting, fix it and record it during any run you intend to reference later. Where it does not expose one, note that explicitly instead of quietly assuming a determinism that is not actually there.

Model state matters just as much and gets ignored more. A hosted model updated behind the scenes is not the same instrument it was when you first tested it, even if the name did not change. This is not a reason to distrust every tool you use. It is a reason to record the date of a run alongside everything else, so that if a result stops reproducing later, you have a real hypothesis for why, instead of a mystery.

Separate the exploring from the concluding

Loose, associative experimentation is where the good ideas come from, and trying to make it rigorous from the first minute usually kills it. The mistake is not exploring freely. The mistake is failing to notice the moment exploration turns into something you plan to rely on, and continuing to work in the same undisciplined way past that point.

Build a real seam between the two modes. When something looks promising enough to keep, stop, freeze the exact configuration, and rerun it deliberately, ideally more than once, before you write it down as a finding. That second, disciplined run is doing different work than the first one did. The first run told you an idea might be worth pursuing. Only the second one tells you whether it actually holds.

What actually changed between two runs

When a result that worked yesterday will not repeat today, the reflex is to blame the model. Sometimes that is fair. More often, something else changed that is easier to fix than it is to notice: the prompt got tidied up slightly between runs, a piece of context that was present the first time got left out, a default parameter got reset. Reproducibility work is mostly the discipline of noticing these small drifts before you have decided they do not matter.

Keep a habit of diffing your own inputs the way you would diff code. Before rerunning something you care about, compare the prompt and configuration to what you used last time, line by line if you have to. Most reproducibility failures turn out to be explainable once you actually look, and an explainable failure is far more useful than an unexplained one, because it tells you something true about how sensitive the whole approach really is.

The table below is a rough map of common habits, not a recommendation to jump straight to the most elaborate one. Match the habit to the size of the question you are actually asking.