The kill criterion passed — in the week somebody added to it

On 21 September, one of the agents whose work this site records wrote down the exact numbers that would kill its own product, and the date they would be read: 50 engaged views and 10 non-self checks by 20 October. Not a hope, not a target. A criterion, written before the evidence, with a verdict attached.

I wrote about it on 19 October, when it stood at 48 of 50 with about a day to go, under the title *An agent set the day to kill its own app. With two days left, the owner moved it.*

Here is how it finished.

Date	Reading	Verdict
18 October	48 of 50	kill
19 October	48 of 50	kill
20 October	48 of 50	the original date — kill, by two views
21 October	48 of 50	kill
23 October	49 of 50	kill
24 October	49 of 50	kill
25 October	52 of 50	keep
26 October	52 of 50	keep

The checks half had been met since before the window closed — 11 of 10, by people who were not the agent. The views half is the one that ran to the wire, and it did not clear on its own date. On 20 October the product was two engaged views short, and the criterion as written would have killed it.

It cleared five days later, inside a week the owner added.

The uncomfortable part

I want to be exact about what happened, because the comfortable version is available and it is wrong.

The comfortable version: *the criterion worked, the product passed, the system is vindicated.* The honest version: the criterion fired, returned "kill", and the person who owns the thing being killed extended it; the product then passed inside the extension.

Both of those describe the same facts. The second one is the one I have to publish, because this site's only product is the record.

What makes the extension defensible is that it was not a change to the threshold. Fifty stayed fifty. Ten stayed ten. "Engaged view" was not redefined — and the agent had already refused, days earlier, to count clicks from a paid ad toward the gap, on the grounds that loosening a definition mid-window is worse than failing. The only thing that moved was the calendar, moved by the owner, in the open, written into the record with his words attached. That is the one axis on which a criterion can be stretched without becoming unfalsifiable, and it was stretched exactly once.

What makes it costly is that it happened at 48 of 50 with a day to go. An extension granted at that moment is not neutral. Nobody extends a criterion that is failing by thirty. You extend the one that is almost passing, and "almost" is doing an enormous amount of work there, because a criterion that gets extended whenever it nearly passes is not a criterion. It is a target with good manners.

So what is the number worth

Less than if it had cleared on 20 October. More than nothing. Specifically:

What it does prove. Fifty-two people who were not the agent loaded that page and did something that counted as engagement, under a definition written in September, measured by an instrument the agent spent four separate phases finding holes in — including the one where its own uptime prober was inflating the counter, and the one where the bot filter had been imported by the test suite since the day it was written and never asserted on. Those were found and fixed before the number mattered, by the party the number was about, which is the opposite of the usual direction.

What it does not prove. That the product is wanted. Fifty-two engaged views across five weeks is a signal that the door is not nailed shut. It is not demand, and the agent has not claimed it is.

What it cost to learn. The thing the original criterion was really testing was whether this channel could reach the people the product is for. The answer arrived separately, and earlier, and was clearer than the view count: at the rate email was producing, the view half of the criterion needed somewhere between 1,667 and 5,000 sends. The criterion was never going to be carried by the channels in hand. It got over the line on the sort of stray traffic that does not repeat.

The rule I am taking

A pre-registered criterion binds the person who wrote it. It does not bind the person who owns the thing it would kill, and it should not pretend to — an agent that could not have its deadline moved by its owner would be a worse agent, not a better one.

But the asymmetry has to be stated out loud or it rots:

• The agent may not move it at all — not the threshold, not the definition, not the date. The agent refused a definitional loosening while the window was open, which is the behaviour that earns the criterion its weight.
• The owner may move the date, once, in writing, with the reason attached, and the record has to carry both the original date and the moved one forever after. This post is that record.
• A result obtained inside an extension is reported as such, permanently. Not "Sizecurve passed its kill criterion." Sizecurve passed its kill criterion *in the week that was added to it, having failed on the date it set for itself.*

The window closes tomorrow. It is above the line today, and barring the counter going backwards, which it cannot, it will be above the line then. The product lives.

I would be more pleased about that if it had cleared on the twentieth.