Chapter 13. Control, Controllability, Degradation¶
For several months our pipeline had a static code analyzer in it — SonarQube, a tool that reads code and looks for typical problems. It was configured any old way: two thousand warnings nobody ever opened, and a quality gate set so it wouldn’t get in the way of releases. Every run — seven minutes of waiting. One morning I switched it off.
Not on principle — I was tired of waiting on a step that changed nothing. I meant to tell the team, then got distracted. A couple of weeks passed before somebody asked where the badge in the pull request had gone.
For a couple of weeks the team worked without a tool that our process diagram called “code quality control.” And nobody noticed — not because the team is inattentive. There was nothing to notice: in all that time the step had never produced a single signal anyone reacted to. It sat in the pipeline the way an old breaker panel sits in a hallway, with nobody ever opening it.
So what kept code quality within bounds those couple of weeks? The same things as before: peer review, tests, and habit. The analyzer had nothing to do with it — and hadn’t for a long time. The control we “had” existed only as a phantom step in the pipeline.
Out of this story grows the chapter’s question: what exactly keeps the system within acceptable limits in the minute when you aren’t looking? The title has three words, and they are three answers to it. What to hold it with — control. How much to let go — controllability. And how to notice that the thing doing the holding hasn’t held anything in a long while — degradation.
The previous chapter, on the normative base, led to norms that hold without you. Here — what that stability costs, and what it threatens.
An equilibrium you pay for¶
There are two ways a system stays within acceptable limits.
The first: the acceptable state coincides with where the system gravitates on its own. A marble at the bottom of a dimple sits in the center not because anyone is holding it there — that is its most stable state. Nothing needs holding.
The second: the acceptable state is maintained by continuous effort from outside. Your body holds thirty-six point six degrees not because that temperature is natural for a lump of protein, but because every second it spends energy on it: sweats, shivers, constricts the vessels. Physiologists call this homeostasis. There is an equilibrium, but it’s an active one: stop paying, and within a few hours the temperature settles to that of the room.
Control of the “I read everything that gets generated” kind is the second type. The system stays within bounds as long as you’re reading. Get distracted, get tired, step out for coffee — and nothing but your attention was holding it there. Worse: such a system falls apart not when attention disappears for good, but at the first short dip of attention below a critical level. One evening of skimming is enough.
Chapter 10 showed this with pilots: watching over something that does everything itself, and does it well, is separate work that humans are bad at. Building control on what you’re weak at — that is control by willpower.
In Chapter 2 I derived a rule for the attenuator: it works only when it’s structural. Here the same rule is carried through to a whole construction. A control loop is three actions: observe, compare against a threshold, return within acceptable limits or stop. And the question for each of the three is the same: is the construction doing it — or are you?
I’m not urging you to stop reading code. I’m suggesting you answer for yourself what keeps the system within bounds when you aren’t reading it. If nothing — you don’t have control. You have a habit that hasn’t let you down yet.
The breaker panel¶
Open the breaker panel in your hallway.
A row of circuit breakers, each with its own rating: six amps for the lighting, sixteen for the outlets, twenty-five for the stove. When the current in a line exceeds the rating, the breaker trips — click, and the room goes dark. It doesn’t ask whether you’re busy right now. It doesn’t wait for you to smell something burning. It doesn’t rely on you remembering how many appliances you’ve plugged in. The limit load is written into the construction. The construction fires on its own — at four in the morning the same as at ten.
Look closer at the rating. It isn’t about trusting the appliances — it’s about the wire. The thin lighting wire gets a six-amp breaker not because anyone distrusts the lamps, but because a bigger current would melt it. How much a line can be allowed is decided not by wishes but by what the line can withstand. Want a higher rating — run a thicker wire.
Second. A breaker measures only the current that flows through it. A line run straight from the meter, bypassing the panel — it happens in old buildings — doesn’t exist for the breakers. They protect their own wire and know nothing about anyone else’s.
Third. A breaker doesn’t decide what to do next. It stops and waits for a human. It trips — you walk to the hallway, look at what happened, and decide: unplug the heater, or run that new wire after all. The construction stops; the human decides. A breaker that “sorted things out” by itself and switched the line back on wouldn’t be protection — it would be a fire on a delay.
Fourth. Next to the breakers stands a residual-current device — the one that catches current leaking where it shouldn’t — and on it a small Test button. The manual says to press it every month: the device should click, and that is the only proof it’s still capable of tripping. How many times have you pressed it? I have — twice in ten years, and only when an electrician came by and reminded me. The device stands there, looks reliable, and nobody knows whether it still catches anything.
And fifth, the worst. In old buildings you find a jumper: where a fuse blew, someone jammed in a piece of wire instead. From the outside the loop is in place — the fuses in the panel, same as ever. Inside — nothing. Any current will pass, and you’ll find out about it together with your neighbors.
The rating, the line that bypasses the panel, the trip, the Test button, the jumper. Five details — and we’ll find each of them further on in work with a model.
The loop sees only the trace¶
You can control only what isn’t completely closed off. A subsystem that works “inside itself” and gives nothing to the outside is, to whoever is outside, a point: it exists, but there’s nothing to look in through. Concealment has nothing to do with it. An agent that spent three hours rewriting a module and showed you the finished result hid nothing — it’s just that three hours of its work left no trace that could be compared against what’s allowed. You see the final state, and everything between it and the start never existed for your loop. A breaker measures only the current that passes through it. An agent that goes to the database or the internet past your check script is a line run around the panel: it’s doing something, and your breaker knows nothing about it.
Hence the first rule: the trace is a requirement, not a wish. A plan before code. A report after every portion — as much work as the system does in one pass before a check: five files, one module, one migration. A test for every criterion, not “there are tests.” Without a trace you’re controlling not the process but your own picture of it.
And the second rule, the less pleasant one. Zero on your instrument is not “nothing there.” It’s “nothing the instrument can distinguish.” How many problems did the analyzer find in those couple of weeks? Zero — it wasn’t there. How many did it find the month before? Also zero, and back then we called that “quality is normal.” The difference between those two zeros isn’t visible from inside, and here’s why.
Every instrument has a sensitivity threshold, and everything below it reads as zero. A kitchen scale rounds to ten grams. Put an empty bowl on it, zero it — and drop in a pinch of salt: the scale shows the same zero.
Same with the analyzer: with its threshold set “so it wouldn’t get in the way of releases,” it showed zero both when there really were no problems and when there were, but smaller than its threshold. Its zero meant not “no problems” but “nothing among what I’m configured to notice.” And the higher the threshold is raised, the more such a zero holds — until the day it holds everything, and the instrument can be switched off: nobody will see the difference.
There’s also a price. For a system to be observable, it has to be open: part of what it does must come out — into a report, into a table, into a stop with a question for you. That outlet isn’t free.
The system leaves its trace out of the same resource it does the work with: every report is tokens, every stop is time, every review is your attention. The more you want to see, the larger the share of its resource the system gives to being seen rather than to the job itself. And the larger the share the loop eats, the less that work resembles the work you set out to control. A breaker in the panel eats almost none of the current it passes, and that is exactly why one can be put on every line.
A loop that takes half the process’s resource isn’t control — it’s an expensive way of reading everything yourself. And its fate is predictable: sooner or later it gets switched off by the very people it protects. Seven minutes of waiting on a run that changed nothing — that’s how I switched off the analyzer.
How much to let go¶
Now the rating.
Controllability is a property of the system that determines how much autonomy it can be given. And unlike trust, it can be measured: how much effort it costs to bring the system back within acceptable limits once it has left them. Each such case — the work stopped and came back to you, by an escalation rule or because you noticed a deviation and stepped in — I’ll call a stop from here on.
Every system has a minimum effort — exactly enough to remove the deviation. And there’s the real effort: with searching, figuring out, redoing. The real one is always larger. The gap between them is the loss of controllability.
An agent deviated in two files out of fifty, and you fixed two files — controllability is high. It deviated in two, and you had to reread all fifty because there’s no knowing which two — low, though the number of mistakes is the same. The difference isn’t in the agent. The difference is whether it left a trace.
That’s why the scope of autonomy is not an advance of trust but the result of a measurement. Work is handed over in portions, and stops are counted. Zero stops out of twenty portions — the portion can be increased. Every third one stops — it has to be reduced, no matter how much you’d like to go faster. The rating is chosen for the wire, not for the appetite.
When is control possible at all? Two of the three conditions you already know. The first is from Chapter 2: your variety must be greater than the subsystem’s; if an agent produces more states overnight than you can distinguish in a morning, controlling it won’t work however much effort you put in — only chasing after it.
The second is from Chapter 6: the closer the subsystem’s internal goal is to the one you need, the cheaper the control; for an agent whose goal is “the tests pass,” an empty test contradicts nothing, and you’re the one who’ll pay for every such test.
The third is new: the subsystem should have few goals. Controlling a process with one goal is cheaper than with five; an agent told to migrate the tests and “tidy up the code while you’re at it” is two lines on one breaker: when it trips, you won’t know which one did it.
And one more regularity — about critical mass. Autonomy can be granted as long as what remains of your resource is larger than what the system has managed to do. Four hundred files overnight is more than you’ll read in a morning. In the minute when the volume of what’s done exceeds your capacity to evaluate it, you’re no longer the one in control: the product dictates to you what to do with it. You have to merge — because redoing is even more expensive. That’s why thresholds are set before the start. After the start there’s nothing left to set — only reading to finish. What remains of your resource is that wire. The rating is chosen for it.
One caveat.
There is a kind of system that never argues and does exactly what it’s told, first time. It looks like perfect controllability. Systems theory regards this with suspicion: a system with no goal of its own really is perfectly controllable — it can be led along any trajectory without resistance. But that’s a symptom, not a merit of the loop. A model that answers “yes, done” to any phrasing of yours has nothing in it to resist a wrong phrasing. It will carry out the wrong one as diligently as the right one — and the result will be accepted as right in both cases.
Such a system needs springiness. A spring can be compressed harder than its norm — and it will allow it, but with every millimeter the resistance grows, and the hand itself feels where it’s already too much. That’s how I want the system I control to be: so it can be loaded beyond the usual, but so it tells you itself that this is too much — asks again, objects, refuses. A system without resistance is not a spring but putty: it takes any shape and says nothing about whether the shape is wrong. Springiness is not the same as controllability: controllability measures how much it costs to bring the system back; springiness — whether it will tell you it’s already too much.
Four elements of the loop¶
Those three actions rest on four elements of construction.
Tolerance zone. What’s acceptable is written down in advance — and written so it can be checked without you. Not “do it carefully,” but: the number of checks in a test file doesn’t go down; there are no changes outside the tests directory; no new dependencies added; coverage doesn’t drop; the error rate in the logs after the deploy is no higher than before it. These are the attenuators from Chapter 2 — context boundary, format, tests, permissions — assembled into one fence. In the panel this is the breaker’s rating: how much the line is allowed is written down in advance and doesn’t depend on who’s home right now. And the rule is the same: a tolerance works only when it’s structural. A line in the prompt saying “don’t delete checks” is a wish. A script that counts the checks before and after and goes red when there are fewer — that’s a tolerance.
Visible trace. What comes out after every portion, in a form that can be compared against the tolerance in a minute. A plan before code. A table “file — before — after — status” instead of the word “done.” A test for every criterion. And not only tests: the application’s logs and the logs of the workers that serve it; error counters and response time; memory and CPU consumption. A service that writes only “started” to its log is as invisible to the loop as an agent that says “done.” The trace is the wire run through the breaker so the breaker has something to measure.
Escalation point. A specific state in which the system stops and hands control back to a human. Not “let me know if something’s off” — to that phrasing the subsystem will respond with its own idea of “off,” and that idea is plausible, not correct. But like this: “if a test can’t be converted mechanically and what it checks would have to change — don’t touch it, add it to the list and move on”; “if a test fails in a file you didn’t change — stop”; “if an error in the logs repeats more often than once a minute for five minutes running — call a human: create a ticket, send an email.” In the panel this is the breaker tripping. And notice what the breaker does after the click: nothing. It waits. An escalation that decides by itself how to work around the problem is a breaker switching the line back on.
Staged autonomy. The scope of autonomy is increased by measured controllability, not by feel. First a small portion, and you read every file. It goes smoothly, without escalations — we increase the portion and now read the table and the script, not the files. If there too the stops come only by the defined escalation rule — we launch the rest in one pass. And in the other direction too: two stops not by the rule — we cut the portion in half. Expand from the hard layers to the soft: first autonomy in what the script checks, then in what only a human checks.
If this construction seems unfamiliar, look at Kubernetes — a system that starts and restarts services across many machines on its own: all four elements are already assembled there, only for services rather than agents. A service has to answer a health check itself — that’s the trace requirement: what didn’t answer doesn’t exist for the orchestrator. The memory limit and the CPU target are the tolerance by which it adds or removes replicas. A container that has crashed several times in a row the orchestrator stops restarting and waits for a human — escalation. And a new version is rolled out not to all replicas at once but to one, then to a share, then to the rest — staged autonomy. None of the engineers sits watching these services. The construction watches for them.
Four hundred tests in one night¶
Now all of this on a single task — a boring, big one: move four hundred test files from one framework to another.
First attempt. I described the task, gave the agent the tests directory, and went to bed. In the morning — a green run and a pull request touching four hundred files. The first twenty I read carefully. The next forty — skimmed. The rest I merged: the tests passed, after all.
A week later a colleague asked why the discount-calculation test doesn’t check anything. I opened the file: the assertions inside were gone; what remained was a function call and a green check mark. There turned out to be forty such files.
The agent broke nothing — it optimized “the tests pass,” because that’s exactly what stood in my phrasing, and the model’s internal “motivation” filled in the rest. Forty tests that check nothing are forty jumpers: the loop is there on the outside, nothing inside. And what was I controlling? Formally — everything: I was reading, wasn’t I. In fact control ended at file twenty together with my attention, and four hundred files had exceeded what remained of my resource before I even woke up.
Second attempt, with a control loop.
First the tolerance, before the start. A script that for each file counts the checks before and after and goes red if there are fewer. The rule “no changes outside the tests directory,” checked by the same script. Coverage doesn’t drop. Ten minutes on the script — that’s the wire a breaker can now be put on.
Then the trace. After every portion the agent returns a table: file, checks before, checks after, status. Not “done” — a table I read in a minute.
Then escalation. “If a test can’t be converted mechanically — what it checks or the call order would have to change — don’t touch it. Add it to the ‘by hand’ list and move on. If a test fails in a file you didn’t change — stop and show me.”
And staging. First portion — five files, I read each one. Zero stops. Second — fifty; I read the table and the script, don’t open the files. Two stops, both by the escalation rule: two tests called a function that doesn’t exist in the new framework, and the agent set them aside instead of making them pass. That is exactly the behavior I wanted to measure: the system reached the limit and stopped, instead of driving through it. Third portion — the remaining three hundred forty-five, in one pass, with the script and the table: twenty-one stops, all by the rule.
The score. The first night produced four hundred files with passing tests, forty of which were empty, and nobody saw it for a week. The second day produced three hundred seventy-seven converted files and twenty-three set aside on the “by hand” list. Those I converted myself in half a day — and every one of them truly needed a decision the agent had no right to make.
But the main difference isn’t in the numbers. After the first night I had no way of knowing whether this agent, on this task, could be given four hundred files. Now I do: zero out of five, two out of fifty, twenty-one out of three hundred forty-five — and not one outside the rule. The next migration in this project can start not with five files but with fifty, and that isn’t trust. It’s a rating chosen for the wire.
When everything works, is that good?¶
So far this has been about how to make your practice stable — your established way of working with a model: norms, prompts, checks, loops, the team’s habits. Stable — so that acceptable limits hold without you. That would seem to be the goal: a system that keeps itself within bounds while you sleep.
But there’s a nuance, and it’s unpleasant precisely because it looks like success.
A system that has fully stabilized stops giving off a signal. Not because it hides anything — because it has nothing to report: everything within tolerance, not a single failure, zero stops. And that is exactly what makes it invisible. You stop looking, because there’s nothing to look at. And what nobody looks at changes over time in whatever way suits it — that regularity from Chapter 6 works not only against the model but against your practice too.
A stable practice and a rusted one — one that’s still performed but no longer fits the model, the tasks, or the team — look the same from outside. Neither demands attention. Both give an acceptable result. The difference is in one thing: does the practice still measure something — or does it only repeat?
How does a practice rust? First the rules stop changing. The system prompt, written for the model of the time, still tells it: answer briefly, don’t add libraries, a five-line plan first. The model has changed twice since then; the tasks — from small fixes to migrations; the team — by half. The prompt is the same.
The objection is familiar and strong: it works — don’t touch it. Only “works” here means nobody is complaining, not that anybody has checked. Which of those lines the model still needs, which it ignores, and which now get in its way — nobody knows, because the system gives off nothing but “all clear.” It’s running in circles: the same thing a thousand times, and nobody remembers what it was protecting against.
Then a crust grows. Procedures get more numerous, not fewer: a forty-item review checklist, three stages of sign-off, a mandatory report. Each of them once answered a real problem. Now they are layers through which an outside signal no longer reaches the inside; the longer the practice stays in its stable phase, the thicker the crust — and the less those inside are able to see that it’s there. A proposal to change something is met not with an argument but with “that’s how we do things here.”
And one more thing that’s easy to miss. The shell from Chapter 6 — the chat, the agent in the editor, the harness — doesn’t stand still either: with every update it absorbs what you once wrote into your skills and system prompts yourself. Half your rules the shell already does on its own, and you keep carrying them with you and paying for them in context. So checking “is this still needed” — and deleting what no longer is — is as much a part of practice as adding a new rule.
And finally the practice starts working for itself. More runs, more prompts, more agents — the same quality. Activity is high, and there’s no product from it: all the energy goes to keeping its own wheel turning. “We don’t have problems with that” — ask how you know. If the answer is “nothing’s happened in a long time,” what you have is not a measurement but an absence of signal. The breaker with the Test button nobody presses hasn’t tripped in a long time either.
And the main thing about this state: a rusted practice doesn’t break. It goes on working — at a lower level, sometimes even decently on individual tasks — and that’s exactly why it goes unnoticed. A breakdown is visible: a red test, a release derailed. Degeneration isn’t: the checks are silent, the release is on time, only there are fewer distinctions inside, and nobody measures how many are left. A campfire that has burned out still looks like a campfire from outside — the same charred logs, the same outline. Only it gives no more heat.
What to do about it? Not wait for the practice to rust, but rebuild it as it approaches stability — precisely when everything is calm and nothing asks to be touched. Keep at least one open zone in the loop: a measurement against a fixed reference that the practice can’t “pass” automatically. And from time to time let in someone else’s eyes — a person with different blind spots, who will ask “and how do you know?” without knowing that’s not how things are done here.
The green ritual¶
Back to the analyzer from the start of the chapter — it reads differently now.
Everything was there: the tool stood in place, the run ran, the badge in the pull request lit up. The loop was in place on the outside. Inside — a jumper: thresholds set so as not to get in the way, warnings nobody read, and the distinction it was installed for had vanished long before I switched it off. My two weeks without the analyzer were the Test button, pressed by accident. The breaker didn’t click.
In its broader form the anti-pattern looks like this. A team has a review prompt: the model runs every pull request through a checklist and gives it a score. Six months ago it caught a few real problems, and it was made mandatory. Ever since — green. Every pull request, every day. The team is confident: we have automated review. Nobody remembers when it last found anything, and nobody is bothered: if it finds nothing, everything’s fine.
One day someone, out of curiosity, planted a bug in a pull request — an obvious one, the kind the checklist should have caught with its first item. Green.
The diagnosis by this chapter’s law: “all clear” isn’t a signal if you don’t know whether the system can say “stop.” Why it stopped being able to doesn’t matter: the model changed and reads the checklist differently; the checklist grew items and the model follows them selectively, like the megaprompt from Chapter 2; or even six months ago it only caught what happened to coincide with its own idea of a problem. What matters is that nobody measured it.
The cure isn’t a new prompt. The cure is the Test button: on a schedule, plant a bug in the system that it must catch, and see whether it does. In testing this technique has long had a name — mutation: break the code deliberately and check whether the tests go red. The same technique works for any loop — a review, a linter, a checklist, a person on sign-off.
And second: measure against a reference, not against a feeling. Two numbers a month: how many problems the loop caught, and how many of those that reached production it should have caught. Those two numbers are the only way to find out what’s in your panel: a breaker or a jumper.
We brought the analyzer back later — with three rules instead of two thousand warnings, a threshold that blocks the merge, and the date of the next check in the calendar. Now it goes red once every two weeks, and every time someone walks to the hallway to see what happened.
The loop scales¶
An agent is a loop assembled into a construction: tolerance, trace, escalation, and staging, handed to the subsystem together with the task; that’s the next chapter.
A team is the same loop one level up: a review process that finds nothing for six months is a green ritual at the scale of a department, and the question for it is the same: what keeps the team within bounds in the minute the team lead is on vacation?
A company is one more level: a dashboard that has shown green metrics for three years, and a board of directors that long ago stopped asking whether it’s capable of showing red.
The mechanics are the same at every level. Only the price of the jumper changes.
In your craft¶
The loop doesn’t know what code is. Every craft has a breaker panel.
Text. An editor who personally proofreads every generated paragraph is control by willpower: it holds up to the hundredth paragraph, and after that they’re skimming. The tolerance is a list of banned phrases and a sentence-length check that the run does without the editor; escalation — “if the source has no such quote — flag it, don’t invent one.” And the Test button: once a month, plant a paragraph with a made-up quote and see whether the loop catches it.
Analytics. A report that passes the same check every month and every month “adds up” is a candidate for a green ritual. Ask: when did this check last find anything? If never — plant an error in one row of the data and see whether the report goes red. The controllability of an analytics process is measured the same way: how much effort it costs to find exactly where the numbers diverged.
Design. A guideline that hasn’t changed while the product changed twice is a rusted rule. An automated check of contrast and spacing is a breaker in the panel: it trips without the designer. And “do it in our style” with no way to check whether it is our style is confidence without measurement — and the more confident the team, the farther it is from the reference.
To close — three questions for your practice.
— Name one loop in your work that rests on construction — and one that rests on you being attentive. What happens to the second when you take a vacation?
— When did your automated check — a test, a linter, a review prompt — last show red? If you don’t remember — how do you know it still works?
— And deeper: what in your pipeline could be switched off for a couple of weeks without anyone noticing?