Evals and Drift

This is the most advanced chapter in the guide, and it is optional. You do not need it to have a working Cortex. You need it the day you rely on an automation enough that a quiet drop in quality would actually cost you. Until then, skim it and come back.

Everything before this chapter made the system exist. This chapter is about keeping it good over time. Left alone, any AI setup slowly slips: files get messy, and outputs quietly get worse. This chapter is how you see the slip coming instead of finding out from a customer.

Why this matters (once it matters)

Here is the failure it prevents. You tweak a prompt to fix one bad output. It fixes that one and quietly breaks three others you did not check. Do that for a few months and the system rots, not because anyone was careless, but because there was no way to see a regression. The point of this chapter is to make quality visible, so a change that helps is obvious and a change that hurts gets caught the same day.

There are two different things slipping, and you catch them in two different ways.

Messy files (structural drift). Broken links, stale dates, two files that disagree. Caught automatically by the checks you already have (the lint commands from the earlier chapters).
Worse output (behavioral drift). The drafts stop sounding like you, the summaries miss the point. You cannot lint that. You have to score the output against a standard you set.

Structural drift is “is the system tidy.” Behavioral drift is “is the system still good.” You want both.

The easy half: automatic checks

You already installed most of this. The lint commands (/vault-lint for your files, /kb-lint for your wiki) scan for broken links, stale dates, and contradictions and hand you a list. This chapter just asks you to treat that list as a baseline you watch over time, not a one-off. Run them on the weekly rhythm from the rituals chapter, and structural drift never gets far.

That is the cheap half, and for many solo operators it is enough. The next half is more work, so only do it where a mistake would actually hurt.

The harder half: scoring the output

To know whether the AI’s output is still good, you compare new output against examples you have judged good. Three simple pieces.

1. Save a few good ones (the golden set)

Every time an automation produces something genuinely good, save a copy in one folder. Ten to twenty is plenty. This is your definition of “good” for that one job. Without it, “better” has nothing to point at.

2. Write down what makes them good (the rubric)

List the few things a good output has, each scored one to five, with a plain description of what a 1 and a 5 look like. For a video description that might be: sounds like you, names the real hook, right length, ends with a clear call to action. The rubric is what lets two people (or two Claude sessions) score the same thing the same way.

3. Score before, change, score after

This is the whole payoff. Before you change anything, score your saved examples. Make the change. Score again. If the number went up, keep the change. If it went down, throw it out. The score decides, not your gut.

saved good examples  ->  score them (say, 4.4 average)
   change the prompt
saved good examples  ->  score again
        4.6  ->  better, keep it
        4.1  ->  worse, undo it

Watch the trend, not any single score. One score is noise. A slow slide from 4.6 to 3.9 over six weeks is the early warning that something upstream went stale.

Pitfalls (the “if this, then that” guide)

Situation	What to do
You want to score everything	Do not. Pick the one or two automations you lean on most. A golden set is real work; spend it where a mistake would actually cost you.
You have no good examples saved yet	Start now. Every time an output is genuinely good, save it. The set builds itself from real work; you do not write it from scratch.
Your scores feel inconsistent run to run	The rubric is too vague. Anchor each score with a concrete “this is a 1, this is a 5” so it is not a guess.
You only look at the latest score	The single score is noise; the trend is the signal. Read the log, not the last run.
The tidy-file checks pass, so you assume you are fine	Tidy files do not mean good output. They are different kinds of drift. Check both.
This feels like overhead with no payoff	It has no payoff until the first time a score catches a regression you would have shipped. Set it up on one automation and wait for that moment.

A worked version: Maya’s

Maya films a lot of short videos, and the descriptions matter for how they get found. So that is the one thing she scores. She saved her ten best-performing descriptions into a folder, and wrote a four-line rubric: sounds like her, names the hook in the first line, right length, ends with a nudge to follow.

When she wanted to change how Claude drafts them, she scored the ten old ones first, made the change, and scored again. The new version dropped on “sounds like her,” so she undid it. She never scores her store emails or her client notes. One automation, the one that matters, measured. That is the whole discipline: not everything, just the thing you cannot afford to have quietly get worse.

You’ve got it when

Your automatic checks (/vault-lint, /kb-lint) run on a schedule and you treat the results as a baseline
At least one automation has a folder of good examples and a short rubric
You have scored one real change before and after, and made the keep-or-undo call from the number
You watch the trend over time, not just the last score

You do not need all of this to have a working system. You need it when the cost of quiet decline is higher than the cost of measuring. For most people that is one or two automations, not the whole system.

The checks named here (/vault-lint, /kb-lint) are in the starter kit and listed in ../03-habits/01-skills-and-commands.md. The scoring half is mostly a folder of examples and a short rubric, not a command.

→ Next: ../99-where-this-goes-next.md