Experimentation Governance for Coaches with Hypothesis Registry and Decision Gates

Most coaches don't have an experimentation problem. They have a memory problem dressed up as an experimentation problem.

You try a new onboarding sequence in Q1. You tweak your pricing page in Q2. You test a different homework cadence with your Tuesday cohort. By the time someone asks "hey, did that thing we tried actually work?" — nobody can reconstruct what was changed, who saw it, what the baseline was, or whether the result was real or just noise. The knowledge evaporates. Six months later you re-run a version of the same test because nobody remembered you already answered that question.

This is the gap between running experiments and having an operational experimentation system. Coaching practices tend to be good at the former and terrible at the latter. Individual tests happen constantly — usually informally, in someone's head — but there's no registry, no shared decision criteria, no way to see the whole pipeline at once. So learning doesn't compound. It leaks.

This article is about the plumbing that makes learning stick: a hypothesis registry, sampling heuristics that work when your cohorts are small, standardized post-mortems, and decision gates that force a real yes/no/kill call instead of tests that just quietly drift into permanence. Think of it as the governance layer sitting on top of the actual test mechanics.

Why experimentation quietly breaks in coaching practices

The failure isn't dramatic. Nothing crashes. It just erodes.

Here's the typical progression. A solo coach starts testing things intuitively — a new intake question, a different session length. It works fine because everything lives in one brain. Then a second coach joins, or the practice starts running parallel cohorts, and suddenly two people are changing things independently. Coach A shortens the discovery call. Coach B adds a pre-work packet. Both notice retention shift. Neither can tell you which change did it because both changes were live at the same time, on overlapping client groups, with no record of when either started.

The real bottleneck across small practices is rarely statistical sophistication. It's coordination. Nobody is stepping on each other's toes maliciously — they just have no shared view of what's currently under test. A registry solves a surprising amount of this before you ever touch a p-value.

The second breakdown is the zombie experiment. A test gets launched, results come in vaguely positive, and instead of a clear decision, it just... stays. The new version becomes the default because nobody killed it. Three of these stack up and now your "standard onboarding" is actually a pile of half-validated changes nobody remembers agreeing to. When retention dips later, you can't roll back cleanly because you don't know which layer caused what.

The third, quieter problem: coaches running tests on tiny cohorts and treating a 6-out-of-10 result as gospel. Small-n testing is legitimate and often unavoidable in coaching — you're not Netflix, you don't have 40,000 users. But it demands different heuristics, and without them you'll chase noise into permanent process changes. There's a whole companion piece on the statistical side in the low-power experiments playbook, and the governance system here is what makes that discipline actually repeatable instead of a one-time act of willpower.

The mistake that comes up most: treating qualitative signal and quantitative signal as interchangeable. With small cohorts, your interview notes and your numbers should agree before you adopt anything. If the metric moved but clients can't articulate why, or clients rave but the metric didn't budge, you don't have a result yet — you have a reason to keep watching. This is also where a solid churn-analysis habit pays off; the interview scripting approach in the churn audit template doubles nicely as a way to pressure-test whether an experimental change actually landed with clients.

The four pieces that make an experimentation system hold together

An operational experimentation system for coaching has four load-bearing parts. Miss one and the whole thing gets wobbly.

1. Hypothesis pipeline — where ideas enter, get shaped into testable claims, and wait their turn. Not every "let's try X" deserves a test slot. The pipeline is a queue with a filter.

2. Hypothesis registry — the single record of everything ever tested, currently running, or killed. This is the memory layer. Without it, learning doesn't compound.

3. Sampling and design heuristics — the rules for how you run tests when cohorts are small, seasonal, or overlapping. This keeps you honest.

4. Decision gates and post-mortems — the forcing functions. Every test hits a gate where someone makes a real call, and every result gets a short structured write-up so the next person can trust it.

These four connect in a loop. Pipeline feeds the registry, the registry tracks the running design, the design produces a result, the result hits a gate, the gate produces a post-mortem, and the post-mortem feeds back into the pipeline as a new idea or a closed question. When one link is missing, the loop breaks and you're back to testing from memory.

The hypothesis registry: your practice's testing memory

Start here, because it's the piece almost nobody has and the one that pays off fastest.

A registry is just a structured list — a spreadsheet is completely fine to start — where every experiment gets one row and a set of standard fields. The point is that anyone in the practice can open it and answer three questions in under a minute: What are we testing right now? What have we already answered? What did we decide and why?

A workable registry schema looks like this:

The single most important discipline: you fill in the hypothesis, primary metric, and baseline before the test starts. The moment you let people record the metric after seeing results, you've built a machine for justifying whatever happened. Pre-registration isn't bureaucratic fuss — it's the thing that stops you from moving the goalposts on yourself.

One pattern worth naming: practices that adopt a registry usually discover, in the first month, that they have three or four "experiments" running that are actually just untracked changes nobody agreed on. That first audit alone is worth the effort.

Sampling heuristics when your cohorts are tiny

Coaching cohorts are small, which means classic A/B math often tells you "you need 900 participants" — useless advice for someone running groups of 15. The answer isn't to abandon rigor. It's to shift what kind of evidence you accept and to design tests that squeeze more signal out of few people.

A few heuristics that hold up in real practice:

1. 1
   Prefer within-client comparisons over between-group when you can. If you're testing a homework format, having the same client experience both formats across two assignments tells you more than splitting 20 clients into two groups of 10. You control for individual variation, which is huge at low-n.
2. 2
   Use pre/post with a stable baseline instead of a control group when a control isn't feasible. A three-month baseline (like the 18% no-show rate above) is often more trustworthy than a control arm of eight people.
3. 3
   Set a "surprise threshold," not a significance threshold. With small samples, decide in advance how big an effect would actually change your mind. If a no-show rate has to drop from 18% to under 10% before you'd act, say that upfront. Small wobbles inside the noise band don't count.
4. 4
   Stack replications instead of chasing one big test. Run the same small test across three consecutive cohorts. Three modest, consistent results beat one dramatic result from a single group of 12.
5. 5
   Watch for seasonality contamination. A retention test that runs November through January will get wrecked by holiday drop-off that has nothing to do with your change. Match your test window to a comparable prior period.

The mistake that comes up most: treating qualitative signal and quantitative signal as interchangeable. With small cohorts, your interview notes and your numbers should agree before you adopt anything. If the metric moved but clients can't articulate why, or clients rave but the metric didn't budge, you don't have a result yet — you have a reason to keep watching. This is also where a solid churn-analysis habit pays off; the interview scripting approach in the churn audit template doubles nicely as a way to pressure-test whether an experimental change actually landed with clients.

Decision gates: the part everyone skips

A decision gate is a scheduled moment where an experiment must get one of three verdicts: Adopt, Kill, or Iterate. No fourth option. "Let's keep it running and see" is not a verdict — it's how zombies are born.

The gate works because it's tied to the pre-registered end date and the pre-registered metric. When the window closes, the owner brings the result to the gate (this can be a 15-minute slot in an existing team meeting for a small practice, or an async review for a solo coach). The verdict gets logged in the registry with a one-sentence rationale.

1. 1
   Confirm the test actually ran as designed. Did the cohort get what was intended? If the design broke mid-flight, it's a null test — log it and re-run, don't interpret garbage.
2. 2
   Check the primary metric against the surprise threshold. Only the pre-registered metric. Secondary metrics are context, not the decision.
3. 3
   Check qualitative agreement. Do client signals match the number?
4. 4
   Assign the verdict. Adopt, Kill, or Iterate.
5. 5
   If Adopt

   write the rollback note. How would you undo this if it turns out to be wrong in three months? Coaches skip this constantly and then can't unwind stacked changes.

6. 6
   Log everything in the registry and close the row.

The rollback note in step 5 is underrated. Adopted changes accumulate, and the ability to cleanly reverse one — because you documented what it replaced — is what lets you keep experimenting without your standard process turning into an untraceable mess.

Standardized post-mortems that take five minutes

A post-mortem here isn't a somber reflection on failure. It's a tiny structured note attached to every gated experiment, win or lose, so the next person can trust and reuse the finding.

Keep it to five fields:

1. 1
   What we tested (one line)
2. 2
   What happened (the metric result vs. baseline, plus the qualitative read)
3. 3
   What we decided (Adopt / Kill / Iterate + why)
4. 4
   What surprised us (the thing you didn't predict — this is where the real learning hides)
5. 5
   What this opens up (the next hypothesis, if any — this feeds back into the pipeline)

That "what surprised us" line does more work than the rest combined. A no-show test might fail on its primary metric but reveal that clients were confused by the calendar invite wording, which becomes a better test than the one you ran. Post-mortems are how a failed experiment still pays rent.

Standardizing the format matters more than the format itself being perfect. When every write-up looks the same, someone scanning the registry six months later can absorb ten past experiments in the time it'd otherwise take to reconstruct one.

A real scenario: the two-coach practice that stopped re-running the same tests

Consider a practice with two coaches and around 60 active clients, running a mix of 1:1 and small-group programs. Before they set up any of this, both coaches were tinkering constantly — new session structures, different check-in cadences, tweaked pricing tiers. Genuinely good instincts, zero coordination.

The concrete cost showed up when they realized they'd both, independently, tested a shortened onboarding sequence about eight months apart — and neither could remember what the first test concluded, so they ran it again. Two testing cycles spent answering a question they'd already answered. Meanwhile their group-program retention had drifted down and they couldn't isolate why, because at least three undocumented changes were live in that program at once.

They set up a shared registry (a single spreadsheet), a monthly 20-minute decision gate, and the pre-registration rule. Nothing fancy. Within about a quarter, a few things shifted. Zombie experiments got surfaced and resolved — two adopted properly, one killed. The duplicate-test problem disappeared because every past question was now searchable. When retention questions came up, they could actually point at which changes were in play and roll one back to test it in isolation.

The measurable win was modest and honest: they stopped losing roughly a full testing cycle per year to re-runs, and group retention recovered a few points once they untangled the stacked changes. The bigger win was harder to quantify — decisions stopped being arguments about who remembered what, and started being lookups.

When this level of governance makes sense — and when it doesn't

When it's worth it:

1. 1
   You have more than one person changing client-facing processes.
2. 2
   You run overlapping cohorts, so untracked changes contaminate each other.
3. 3
   You've caught yourself re-running or re-debating something you already tested.
4. 4
   You're scaling and need onboarding coaches to inherit decisions, not just processes.

When it's overkill:

1. 1
   You're a solo coach with a handful of clients testing one thing at a time. A lightweight note in your KPI review — see how the review cadence works in the coaching KPI dashboard guide — is plenty. Don't build a registry to manage two experiments a year.
2. 2
   You're in a launch crunch and have no capacity to run and govern tests. Better to run fewer tests well than to build ceremony around tests you won't finish.

Who should not do this: A practice that isn't actually testing anything yet. Governance on top of zero experiments is pure overhead. Get into the habit of forming and checking hypotheses first; add the registry and gates the moment you have more than a couple running at once, because that's exactly when the memory starts leaking.

Making it a workflow, not a binder

The whole thing lives or dies on whether it's low-friction enough to actually use. A system that requires ten minutes of admin per experiment gets abandoned inside a month.

The realistic workflow for a small practice runs like this. Ideas land in a running "hypothesis pipeline" list whenever anyone thinks of one — no filtering yet, just capture. Once a month, someone spends ten minutes promoting the two or three best candidates into the registry as pre-registered experiments with metrics and baselines locked. Tests run for their fixed windows. At the monthly gate, closed tests get a verdict and a five-field post-mortem, and adopted ones get a rollback note. The pipeline refills from the "what this opens up" lines. That's the loop.

Whether you run that in a spreadsheet or in a dedicated experimentation platform matters less than the discipline itself. What a proper system buys you as you scale is the automatic stuff a spreadsheet can't do: nudging owners when a test window closes so results don't rot, keeping the registry searchable so nobody re-runs old tests, linking post-mortems back to the specs and runbooks that produced them, and flagging when two experiments are quietly overlapping on the same cohort. Those are exactly the coordination failures that eat small practices alive as headcount grows — and they're worth centralizing once you've got more experiments in flight than one person can hold in their head.

“

Start with the spreadsheet and pre-register experiments before you launch to keep friction low.

Here's a quick diagram of the workflow loop.

That's the loop.

The core idea underneath all of it: experiments are only valuable if what you learn survives longer than the person who ran them. A hypothesis registry, honest small-n heuristics, and decision gates that force a real verdict are how a coaching practice turns scattered tinkering into knowledge that actually compounds. Start with the spreadsheet, pre-register before you launch, and never let a test end without a decision. Everything else is refinement.