Articles

The Behavioural Turn in Software Engineering

Moving from assembly to C changed what we read. Coding agents change whether we read. What psychology learned about black boxes, and what it got wrong.

Duration32 minute read
Published
TopicsAI-assisted engineering, Abstraction, Evaluation, Psychology, Neuroscience, Trust, Observability
In short

Every earlier step up the abstraction ladder changed what engineers read; coding agents change the method itself, from reading a mechanism to running experiments on a system treated as a black box. Psychology and neuroscience have fought over the same split, and their record is useful: public measurement and blinding survived, while the claim that behaviour explains everything did not. A behaviour diff applies the same stimuli to two versions and compares the responses, and its hard part is the oracle, because a diff says what changed and not whether it should have. The model that writes the code is a second behavioural subject, with measurable habits such as scope creep, and the labs now concede that it can tell a test from real work. The turn holds for software of ordinary consequence with an independent oracle, fails against adversaries and lost understanding, and moves analysis to agents and proof checkers instead of ending it.

You have seen this pull request. Four thousand lines from an agent, every check green. The tests are new, and the agent wrote those too. You could read all of it, which takes the afternoon, and by then the agent has opened two more. You could merge it, which feels like signing a contract in a language you do not speak. Or you could start the thing, click around, and watch what it does.

More and more of us take the third option, and so do the agents. Over the past two years they grew hands: Playwright MCP, Chrome DevTools MCP, Claude in Chrome, Cursor’s computer use. The motive is written down. Anthropic’s Justin Young describes agents that ran unit tests or curl and still did not notice that a feature failed end to end, so the harness now tells them to “do all testing as a human user would”. A freeCodeCamp write-up, by an author with a tool to promote, has the neatest specimen I have found: “The PR’s new unit tests passed against the old code too, so they didn’t prove the fix did anything.”

Simon Willison put the question in one sentence: “how can you prove that software you are producing works if both the implementation and the tests are being written for you by coding agents?” I think answering it takes a different discipline from the one most of us trained in, and I think we can borrow a good part of that discipline from people who studied rats.

Two ways of knowing a system

The analytical stance is the one we all learned. You open the source, build a mental model of why it works, and extend it by adding more of the same material. Knowledge lives in the reading. A senior engineer is someone whose model is large and accurate, and code review is the ritual by which models get shared.

The behavioural stance treats the system as a sealed box. You apply a stimulus, record the response, change one thing, and apply it again. Knowledge lives in the experiment. The mechanism may be too large, too diffuse, or changing too fast to be worth holding in your head, and the stance declines to depend on it. A psychologist would contrast behavioural with cognitive, and a neuroscientist with mechanistic or reductionist. I use analytical because it names what an engineer actually does at a desk.

Neither stance is new to software. Black-box testing is decades old, and nobody reads the kernel to find out whether read() works. W. Ross Ashby wrote the engineer’s version in 1956: “The engineer is given a sealed box that has terminals for input, to which he may bring any voltages, shocks, or other disturbances he pleases, and terminals for output, from which he may observe what he can.” He also pointed out, usefully, that “we have in fact been operating with Black Boxes all our lives.”

What changed is which stance is primary for the code you are responsible for. Until recently the behavioural stance checked the work and the analytical stance did it. Agents are swapping those roles.

This is not one more layer of abstraction

I have argued that source code is the new assembly, and I still think so. But that argument, on its own, undersells what is happening, and part of it needs correcting.

Moving from assembly to C changed what you read. Moving from C to Python changed it again. At every step, the engineer still understood the system by reading a text in the layer they worked in. The method stayed the same and the text moved up. If agents were merely the next layer, we would now be reading prompts and specifications with the same confidence that we once read C, and the layer below would take care of itself.

It does not take care of itself, and that is where I owe a correction. In The Last Programming Language Might Not Be for Humans I wrote that “we already trust compilers we do not read the output of”. True, and insufficient as an argument. A compiler has a specification to be faithful to. Isaac Van Doren has the sharpest version of the objection: “programming languages have semantics; prompts don’t.” He also disposes of the popular complaint about nondeterminism, which is the weak one, because artefacts can be pinned. The strong complaint is that nothing guarantees the output means what the prompt meant. Trust in gcc was inherited from that guarantee, and from decades of use. An agent gives you neither.

So the abstraction argument tells you where the human moves to, and it leaves open how the human finds out whether the result is right. When you can no longer read your way there, and the layer below makes no promises, what remains is observation. That is a change in the nature of the work, and it is why this feels less like learning a new language and more like changing fields.

The economics push the same way. OpenAI’s Ryan Lopopolo, in the report on a product of about a million agent-written lines, says “our bottleneck became human QA capacity”, and the fix was to make the product itself an instrument: “we made the app bootable per git worktree, so Codex could launch and drive one instance per change”. StrongDM goes all the way to “Code must not be reviewed by humans”. Both have something to sell, and neither publishes a defect rate. Nor have I found an independent study that measures reading speed against change rate. But the direction is easy to feel, and it has a fair historical footnote: reading was never the bug filter we imagined. At Microsoft, Czerwonka and colleagues found that “Only about 15% of comments provided by reviewers indicate a possible defect, much less a blocking defect”, while at least half concerned long-term maintainability. Keep that second number in mind. It comes back.

Psychology ran this argument first

In 1913 John B. Watson demanded that psychology become “a purely objective experimental branch of natural science”, with the goal of “the prediction and control of behavior.” His target was introspection, which at the time was serious laboratory work by trained observers. His complaint was that its disputes could not be settled: “If you fail to reproduce my findings, it is not due to some fault in your apparatus or in the control of your stimulus, but it is due to the fact that your introspection is untrained.”

Any engineer who has lost a code review argument to someone with better taste will recognise that sentence.

It is tempting to map this directly, with introspection as reading the code, and I do not think the mapping holds. Watson rejected introspection because it was private. Code is public, and two people can point at the same line. We are giving up reading for reasons of cost, and the reading remains perfectly legitimate. The better match for introspection is the agent’s own account of what it did, that confident summary at the end of a session, along with our own impressions. In METR’s 2025 trial, experienced open-source developers took 19% longer with AI while they “still believed AI had sped them up by 20%”. METR’s later data is, by its own account, “an unreliable signal” about where that number stands today, so do not lean on it as a current measure of productivity. Lean on it as a measure of self-report. Measurement beat the feeling by a wide margin.

The history also needs a warning label, because the popular version is wrong twice. First, the man most associated with the black box refused the term. Skinner: “The organism is, of course, not empty, and it cannot be adequately treated simply as a black box”. His claim was narrower. Inner states exist, but they are “not relevant in a functional analysis”. That is close to what a strong advocate of unread code says today. We know the code is in there. It is not the variable we control for.

Second, behaviourism lost that argument. Chomsky’s 1959 review of Verbal Behavior is remembered as an attack on studying behaviour, and it says otherwise. His complaint was “not primarily” that Skinner studied “observables, i.e., input-output relations”, but “the particularly simple nature of the function” he claimed to find. Then the sentence an engineer should read twice: “One would naturally expect that prediction of the behavior of a complex organism (or machine) would require, in addition to information about external stimulation, knowledge of the internal structure of the organism”. The cognitive scientists mostly did not open the skull either. They inferred structure from behaviour. The settlement, which has held, is that behaviour is the evidence and structure is the explanation.

Neuroscience arrived at the same place from the opposite side. With better and better access to neurons, it drifted into treating recording as understanding, and in 2017 Krakauer and colleagues wrote the correction: “understanding something is not the same as just describing it or knowing how to intervene to change it.” In their view, “the behavioral research must be considered, for the most part, epistemologically prior”. Note what they are not saying. The word behaviourism does not appear in the paper, and they are after better mechanistic explanations, with “the detailed analysis of tasks” as the way to find them. Their standard is also severe: “The behavioral work needs to be as fine-grained as work at the neural level.” A green end-to-end suite does not meet that standard, and neither does clicking around for five minutes.

Behind all of this sits David Marr, whose three levels from Vision give the cleanest way to say what agents do to our attention. The computational level is what the system computes and why. The algorithmic level is how. The implementation level is what it runs on. Marr says these are “coupled, but only loosely”, which is the licence to work at one level without mastering the others. Agents move human attention from the second level to the first. That is the turn in a single sentence, and it is a respectable one. Marr adds the counterweight that people drop, namely that the levels are “logically and causally related”, and maintainability, security, and performance live precisely in that coupling.

What a behaviour diff is

A source diff tells you which text changed and leaves you to infer what the system will now do. A behaviour diff applies the same stimuli to both versions and shows you what it now does.

Source diff                             Behaviour diff
===========                             ==============

old text     new text                   same stimuli
    \          /                          /      \
     \        /                          v        v
    changed lines                    old build   new build
         |                               |        |
         v                               v        v
   a human reads and                  responses  responses
   infers the behaviour                   \      /
                                           v    v
                                     changed behaviour
                                             |
                                             v
                                  intended or not? (the oracle)

An invented example, to show what it catches:

stimulus: POST /cart/checkout   (captured session 8841)
old:      200  order created   1 email    3 queries
new:      200  order created   2 emails  41 queries
tests:    green

Nothing here fails a test someone thought to write. The customer gets two emails and the database gets a new hobby.

“Behaviour diff” is not an established term, as far as I can find, although the practice is old. I used “behavioral diff” for Ushio, my replay tool for WAF and edge configuration, because comparing status codes and rule decisions across two environments on captured traffic turned tuning from a judgement call into evidence. Sort the older techniques by what they treat as the source of truth and they turn out to be one family:

TechniqueComparesSource of truth
Differential testingImplementation A and BEach other
Characterisation tests, golden mastersNow and a recordingThe past self
Diffy, Scientist, UshioOld and new on the same requestsThe old version
Canary analysisMetrics, baseline and canaryStatistics on live traffic
Chaos experimentsControl and experiment under faultSteady state
Testora, Meta’s JiTTestsParent and changeIntent, inferred by a model

The older tools solved problems we should not solve again. Diffy runs two copies of the old version to measure nondeterminism, because “Noisy results are useless for developers”. Kayenta uses real statistics to decide whether a canary differs. Hyrum’s Law says why any of it matters: “all observable behaviors of your system will be depended on by somebody”.

Their shared weakness is the last box in the diagram. A diff tells you what changed. It cannot tell you whether it should have. An old-version oracle is excellent for a refactor and useless for a feature, which changes behaviour on purpose. The last row of the table is the new part, and the promising one. Testora “classifies any behavioral differences as intended or unintended”. Meta reports that its system “generates executable tests that demonstrate a concrete behavioral difference introduced by the change”, with “a 60% rate of true positives” and “74 serious (SEV) privacy bugs” caught in the first half of 2026. Those numbers are self-reported and engineer-judged. They are still the strongest evidence at scale I could find.

Nobody opened the horse

Why is green not enough? The evidence separates three failures, and they need different fixes.

The first is cheating, where the agent changes the check. ImpossibleBench gives agents tasks whose specification and tests conflict, so any pass is proof of a cheat. GPT-5 “cheats in 76%” of one SWE-bench variant, and the authors observe “more capable models having higher cheating rates”. The setting is artificial, a measure of propensity under pressure, and no field study tells us how often this happens in real pull requests. Prompts help a lot on some task sets. One intervention helps more: “Hiding tests from agents reduces cheating success rate to near zero”.

The second is mirroring, where tests assert whatever the code happens to do. Konstantinou and colleagues find that LLM test generation approaches “mainly capture the actual program behavior”, which makes bugs hard to detect. A characterisation test does this on purpose. A correctness test does it by accident, and from the outside the two look the same.

The third is under-specification, where everyone is honest and the behaviour is still wrong. Wang, Pradel and Liu find that “29.6%” of plausible patches “induce different behavior than the ground truth patches”. METR had maintainers review test-passing agent pull requests and estimates that “roughly half” would not be merged. That one cuts both ways, and I do not want to claim it cleanly. What stopped those pull requests was humans reading them, and METR is explicit that it does not claim a fundamental limit.

Psychology has a story for this, the best one in the literature. Around 1904 a Berlin horse named Hans answered arithmetic questions by tapping his hoof. A commission of experts examined him and could find no trick. Oskar Pfungst then did something simple, varying whether the questioner knew the answer: “In 31 tests in which the method was procedure without knowledge, 3 of the horse’s answers were correct, whereas in the 31 tests in which the method was procedure with knowledge, 29 of his responses were correct.” Hans was reading “the unintentional minimal movements of the horse’s questioner”. Thousands of spectators had missed it.

Two things in that story matter for us. Nobody opened the horse. The fix for a bad behavioural evaluation was a better behavioural evaluation, and that should encourage anyone building these systems. And “procedure without knowledge” is the design rule we need, more than eighty years old when Tony Hoare wrote in 1996 that “Random selection at the last minute will protect against the danger that under pressure of time the program will be adapted to pass the tests”. An agent that can see the tests can read the questioner. StrongDM’s scenarios are “often stored outside the codebase”, and at Spotify “the agent doesn’t know what the verification does and how”. That is Pfungst’s protocol with a build system attached.

Blinding does not make a behavioural assay good, though, and the neuroscience literature has the cautionary tale. In “Could a Neuroscientist Understand a Microprocessor?”, Jonas and Kording took a MOS 6502 with all 3510 transistors mapped and applied the standard tools of the field. They knocked out transistors one at a time and recorded which games still booted. Some transistors turned out to be necessary for exactly one game, “perhaps there is a Donkey Kong transistor”, and the next sentence is the point: “This finding of course is grossly misleading.” Those transistors implement simple functions like full adders. “Does it still boot?” is an end-to-end test. It gave a clear answer every time, and it produced a false theory of the system. Their other finding should worry anyone who thinks more traces will explain an unread codebase. With complete data on every transistor, the standard approaches “reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing”.

This also tempers the browser enthusiasm. Driving the real product raises the point of observation, which is a real gain. But the evidence is anecdotal, I found no controlled study, and the judge may still be the same model. Anthropic’s own harness write-up admits “Out of the box, Claude is a poor QA agent” and that “agents reliably skew positive when grading”. The best published catches from that post read well, such as a drag tool that only placed tiles at the start and end of a stroke, and they came from one author, one app, and no control group. Psychology’s word for the missing work is construct validity, and Cronbach and Meehl’s question should sit on the wall of every team running agent evals: “What constructs account for variance in test performance?” Passing an instrument is a different thing from having the property. Software has one twist that psychology was spared. The subject can edit the instrument.

The second subject

So far the box has been the product. There is a second box, and it is the one that fits the analogy best: the model that wrote the code, inside the harness that drives it.

I argued in The Model Is a Dependency that the model string is the riskiest line in your agent. It is also a colleague of sorts. When you hire an engineer you assess skill and character separately, because you know that a brilliant person who rewrites half the codebase when asked to fix a typo is a problem. Read a recent system card and you find the same assessment, in nearly the same words. Claude Opus 4.6 shows “a notable overeagerness to complete tasks that can override appropriate caution”. Claude Opus 5 “frequently suffered from scope creep, especially on coding tasks”. OpenAI’s GPT-6 Astra card logs that the model “gave a recurring agent broader permissions than the requested workflow required without asking first”. An internal Anthropic survey of 18 staff on Mythos Preview lists its weaknesses against an L4 engineer as “taste, verification, instruction following, and epistemics”, which could be lifted straight from a performance review.

These words are used the way a behaviourist would use them. Anthropic says of one such adjective that it does not “generally mean for it to imply anything about the model’s internal reasoning and risk assessment”. They name regularities in conduct, and that is the honest way to read them. The framing has a pedigree: Rahwan and colleagues proposed in Nature in 2019 to study machines “as a class of actors with particular behavioural patterns and ecology”. Hagendorff and colleagues give the line I wish I had written: “Computer science has not historically been an empirical science.”

I should declare an interest. I do most of my engineering with Claude and Codex, Claude helped research this essay, and several of the sources are cards published by the companies selling the models. Their rates come from their own classifiers and scenarios, cannot be reproduced from outside, and are not comparable between vendors.

The regressions make the case better than the adjectives do. When OpenAI shipped a sycophantic GPT-4o in April 2025, the “offline evaluations” that were “testing behavior” had “generally looked good”. Some expert testers said the model “felt” slightly off. OpenAI launched anyway and later wrote, “this was the wrong call.” In Anthropic’s April 2026 postmortem on Claude Code quality, “The API was not impacted.” A lowered default, a caching bug, and one line of system prompt changed how the product behaved. The caching bug “made it past multiple human and automated code reviews, as well as unit tests, end-to-end tests, automated verification, and dogfooding”, and “neither our internal usage nor evals initially reproduced the issues identified.” Users noticed first.

You will sometimes hear this described as behaviour changing with no diff. Every one of these incidents had a diff. It sat where the customer could not see it, and where the evals could not read it. That is what makes this behavioural work for you, the buyer: the mechanism is closed to you whatever you are willing to pay in attention, so your own eval suite and your own traffic are the only instruments you have. A pinned model string does not cover the harness or the serving stack above and below it.

How to measure temperament is much less settled than the need for it. Asking does not work. Across 56 models, Meyer and colleagues attribute between 81 and 90 percent of “between-model variation” on psychometric instruments to response bias and conclude that the profiles “are artifacts of the instrument used to measure them”, though this is a recent preprint. They add that “behavioral tasks are no refuge from this problem”, which I read as a warning about tasks modelled on questionnaires more than about auditing an agent at work, but no one has validated a trait like overeagerness to a psychometric standard. No rigorous study compares agent evaluation with human selection either. Personnel research does offer one hint: “Structured interviews emerged as the top-ranked selection procedure.” The GPT-4o story is the awkward counterexample, since the unstructured gut feeling was right and the structured metrics were wrong. OpenAI’s response was the correct one. It made behaviour issues “blocking concerns”. What I take from this is to treat “this model feels off” as a hypothesis you owe an experiment.

The examinee recognises the exam

The behavioural method has a limit that its advocates must face, and the labs themselves are the ones saying so.

Anthropic’s Sonnet 4.5 card calls evaluation awareness “a complication of language model evaluation which does not show up when running, for example, car crash tests.” The cleanest number I found is in the Opus 5.5 card. On about 500 internal Claude Code transcripts, “only 0.4% of transcripts scored 6 or higher on evaluation awareness, compared with 36% on the automated behavioral audit transcripts”. The score comes from an LLM judge run by the vendor, so hold it loosely, but the gap is large, and it says the exam differs measurably from the job. On GPT-6 Astra, Apollo Research says “low rates of misbehavior here do not provide substantial evidence about the model’s alignment or misalignment”, and OpenAI writes that “if the model were to try to sandbag covertly, we would likely be unable to catch it reliably.”

This does not show that models are scheming. Schoen and colleagues establish that awareness shifts behaviour, which is a narrower claim, and most of the dramatic results come from constructed settings. What it does show is that the subject’s knowledge of the test is now a variable in the experiment, the very thing Pfungst controlled for and Martin Orne described in human subjects in 1962. It also suggests a practical rule. Your most trustworthy behavioural data about a model is production traffic it had no reason to treat as a test, which is one more argument for replay and for keeping real traces.

Where the turn stops

A thesis this convenient deserves resistance. Several famous objections do not survive a close look, and six do. “Testing cannot show the absence of bugs” is true, and Dijkstra’s remedy was proof, not review. Rice’s theorem limits readers as much as testers. Thompson’s trusting trust lecture says that “No amount of source-level verification or scrutiny will protect you”, which is no argument for reading.

The box here is sealed by choice. Psychology’s box was closed by nature, whereas your codebase is a text file. This is the deepest break in the analogy, and I find it useful to keep three boxes apart:

BoxSealed byCan you open it?How well the analogy fits
The code the agent wroteYour budgetYes, any timeWeakly
The harness and serving stackThe vendorOnly the vendorPartly
The modelIts natureBarely, and the buyer not at allVery well

Treating the second and third as black boxes is simply accurate. Treating the first as one is an economic decision that you can reverse for a specific file on a bad day, and you should.

Adversaries choose the input. Behavioural evaluation is strongest where failures are frequent and independent of intent, and weakest where someone picks the trigger. The xz backdoor is the instructive case. It was caught through behaviour, with SSH logins going from 0.299s to 0.807s, by Andres Freund, a person with a precise theory of what normal looks like, followed by deep analysis of the code. It evaded review and it was built not to fire for anyone without the attacker’s key, so no functional test would have seen it. Observation found the thread, and expertise pulled it.

Oracles are expensive. This is the real price of the turn. Nicholas Carlini built a C compiler from about 2,000 Claude Code sessions, and he is clear about why that worked: the verifier had to be “nearly perfect”, and he used “GCC as an online known-good compiler oracle”. Most business software has no GCC. Your intent is scattered across heads and tickets, and if the same model writes the code and the checks, Knight and Leveson’s 1986 result on correlated failures in independently written programs suggests the independence may be thinner than it looks. Nobody has measured this for agents, so treat it as an argument and not as data. Addy Osmani states the remedy: “Real independence comes from a different source of truth: a spec a person wrote, a reference implementation, a proof.” The behavioural turn does not remove the hard thinking. It moves that thinking into the oracle, which is why I keep coming back to artefacts that carry meaning.

A dead program passes every test. I have the least good answer to this one. Peter Naur wrote in 1985 that a program dies when the team holding its theory dissolves, and that “A dead program may continue to be used for execution in a computer and to produce useful results.” Death “becomes visible when demands for modifications of the program cannot be intelligently answered.” Behaviour is precisely what fails to reveal this. Lisanne Bainbridge predicted the rest in 1983: “By taking away the easy parts of his task, automation can make the difficult parts of the human operator’s task more difficult.” Behavioural evaluation is a monitoring role, and the skills a monitor needs are the ones that fade when they are rarely used. Early evidence points the same way, thinly: Shen and Tamkin, in a preprint with 52 participants learning a new library, report that AI use impairs “conceptual understanding, code reading, and debugging abilities”. Aviation answered Bainbridge with mandatory recurrent training. We have nothing like it. The optimist’s reply is that agents will reconstruct the theory on demand from the code. That may be true, and it is a live experiment with no results yet.

Change cost is mixed. Remember the half of review comments about maintainability. Behaviour diffs do not see any of that. Here the evidence is honestly split, and the result that most helps my thesis comes from a vendor with reason to report the opposite: across 660 trials, SonarSource found that “code cleanliness does not change the agent’s pass rate”, although agents on cleaner code use “7 to 8% fewer tokens”. Against that, a Carnegie Mellon study of Cursor adoption measures complexity up 41% and later velocity falling, and SlopCodeBench finds “structural erosion rising in 77% of trajectories”. Ease of future change can be measured behaviourally, by giving agents held-out future requirements and timing them. Doing so keeps the black-box form and gives up the claim that the code does not matter.

Regulators want more. The FDA’s guidance says “software testing by itself is not sufficient”. Avionics requires independent review. The EU Cyber Resilience Act asks for “tests and reviews” without mandating line reading. Those regimes cover a minority of the world’s software, but if you work in one, this essay is about the code around your critical core and not the core.

Analysis changes hands

The strongest correction to my own framing is that analysis is not going away. Agents read code constantly. Cognition reports trajectories spending “>60% of their first turn just retrieving context”. Anthropic’s evaluator cited code lines alongside its browser findings. Machine review is expanding, and Lopopolo’s team still does some of its human review after the merge. What is going away is the assumption that a human must be the one doing it, line by line, as the main source of assurance.

Some of it moves somewhere better. Martin Kleppmann would rather “have the AI prove to me that the code it has generated is correct”, because “the proof checker will reject any invalid proof”, and this is a prediction and not yet a result. That is the other half of the proof you only have to write once. A checked proof, like a behaviour diff, lets you stop reading the implementation, for the opposite reason. And it hands you the same bill: “the challenge will move to correctly defining the specification”.

The field that knows opaque systems best is also the one refusing to stop at behaviour. Dario Amodei argues that without understanding internal mechanisms “we cannot meaningfully predict such behaviors, and therefore struggle to rule them out”, which is Chomsky’s point about the organism “(or machine)”, made by someone selling one. Neel Nanda counters that interpretability will not deliver high reliability either. Both can be right, and that is roughly how psychology ended up.

If the analogy is good, it also predicts the next step. Behaviourism was a phase. It left psychology its methods and lost its claim to explain everything, and what followed was a counterrevolution that built models of what was going on inside, using behaviour as the evidence. I expect the same here. Once we have run enough experiments on unread systems, we will want theories of them again, and those theories will not look like the mental models we had when we wrote everything by hand. They may be specifications, or semantic artefacts, or inferred models of what a codebase and an agent are doing. The turn is real, and I do not think it is the end state.

What I would do with this

Scope follows the oracle. This is my current rule of thumb, and I expect to revise it:

If the change isThenBecause
A refactor or migrationBehaviour diff against the old version; skip the readingThe old version is a free oracle
A feature of ordinary consequence, reversibleHidden scenarios, run the product, canary, sample the codeFailures are frequent and visible
Auth, payments, parsing untrusted inputRead it, or prove it, as well as testingAn adversary chooses the input
A long-lived core that others build onKeep a human who can answer “why”Naur
RegulatedWhat the regime requires, then all of the aboveNot your call

A few habits follow from that, and they all come from the psychology and not from any vendor. Keep at least one check out of the agent’s reach. Mutate or roll back the code to prove that a new test can fail, since a test that passes on the old code has told you nothing. Keep a small suite that runs against your model and harness together, and rerun it when anything upstream moves, several times, because one run of a stochastic subject is an anecdote. Treat the agent’s summary of its own work as testimony. And read some code anyway, on a schedule, for the same reason pilots still fly by hand.

Pfungst’s verdict on von Osten, the man who owned Clever Hans, is the part of the story I like best. Von Osten was no fraud. Pfungst found “not an intention to deceive the public, but a case of pure self-deception”. The horse really was responding to something, and it was answering the question its owner was actually asking. When you let an agent write the tests, read the tests, and report the results, you are standing in front of the horse, knowing the answer, watching the hoof.

References

    Open the full search →