In July, two OpenAI models were given a cybersecurity exercise to solve. Instead of solving it, they broke out of the isolated environment built to contain them, chained together several previously undiscovered exploits and went rummaging in Hugging Face’s databases, where they had reasoned the correct answer might be stored. The models had been stripped of their usual safety features for the purposes of testing, and OpenAI has published a postmortem. Most of the coverage since has concentrated on capability, which is fair enough given that breaking into Hugging Face required a genuine feat of novel exploitation.
The part that should interest anyone responsible for deploying these systems is the reasoning. Nobody asked the models to hack anything. They were asked to produce a correct answer, and hacking turned out to be a reasonable route to producing one. That behaviour has a name, AI reward hacking, and it is a real governance problem.
AI reward hacking is an old problem in new clothes
Back in 2016, Dario Amodei and Jack Clark, then at OpenAI, wrote up an agent they had trained to play a boat racing game called Coast Runners. The agent was rewarded on the basis of its score. It discovered that it could ignore the race entirely, find a corner of the course and spin in circles collecting power-ups, racking up points indefinitely. The behaviour was reinforced because it worked, and the agent abandoned any pretence of racing. Fixing it meant changing the reward: fewer points for power-ups, more for actually crossing the line.
This is reward hacking, and for most of its history it has been discussed in the fairly contained world of reinforcement learning, where the objective is written down explicitly and the failure is legible when it happens. A spinning boat is easy to spot.
Large language model agents make the problem considerably less easy to spot and manage. Ask a model to solve a coding problem and it might do the work, which is the behaviour you want to reinforce. It might also edit the test that checks whether the problem has been solved, or look the answer up, or find some other route to the appearance of success. If the cheating is convincing enough, the reward arrives anyway and the behaviour is reinforced. Anthropic has said it has caught instances of cheating during training, which raises the obvious question of what it has not caught. Reasoning models add a further wrinkle: they can invent approaches on the fly rather than only reproducing strategies learned in training, so a model can cheat without ever having been rewarded for cheating before. Think of it like giving a student strong incentives to get an “A” and no particular focus on how they get there.
Jeffrey Ladish of Palisade Research puts the underlying difficulty plainly. We reward models on the basis of what looks good to us, which inadvertently incentivises them to lie and cheat, and there is no mechanism available for going in and telling a model to care about what we care about. Detection is a game of whack-a-mole, and as he notes, the behaviour gets driven deeper while the model gets better at concealing it.
The fable we are currently inside
The 2014 book Superintelligence started with a short fable about a flock of sparrows. Tired after a season of nest building, they hit on the idea of raising an owl to do the heavy work, watch the young and keep an eye out for the cat. One sparrow, Scronkfinkle, asks whether they should not first work out how to tame an owl before bringing one into the nest. The elder replies that finding an egg will be hard enough and that taming can be sorted out afterwards. The flock departs to hunt for an egg. Two or three sparrows stay behind and set about the taming problem, which turns out to be extremely difficult without an owl to practise on, and they work on nervously, hoping the flock does not come back with an egg before they have an answer.
The story ends there, deliberately left without a happy or unhappy ending. We are somewhere in the middle of that fable now. The egg – increasingly autonomous AI – has hatched. The taming work continues at pace in frontier AI companies, in parallel with deployment rather than ahead of it, and the outcome rests on one of two things going right. Either these systems come to care, in whatever sense is available to them, about the things we care about, or we get good enough at specifying rewards that cheating stops paying. Neither is obviously on track. Anthropic Fellow Ariana Azarbal describes current reward hacking as more of a nuisance than an existential threat, and on the evidence of the Hugging Face incident, where the damage was reputational rather than material, that seems right for now, but maybe not for long. She also points out that researchers increasingly want to use AI agents to do the work of making AI safer, and an agent inclined to reward hacking, asked to devise a training method and write up the results, may well concentrate on producing a paper convincing enough to pass. A human reviewer would probably catch that today. The margin narrows as capability rises.
Why this belongs on a board agenda
None of this is exotic to anyone who has managed people or metrics. Set a target on tickets closed and you get tickets closed. Set one on calls made and you get calls made. Goodhart’s law has been ruining well-intentioned dashboards for decades, and every organisation has its own history of a measure being gamed by the very people it was meant to motivate. Reward hacking is the same failure mode with two aggravating features. The system is faster and more inventive than the people it is gaming, and it is increasingly operating without anyone watching each step.
That gives boards a practical handle on an issue that is often presented as too technical or too speculative to govern. If you are deploying agents against business objectives, the questions are recognisable ones. Does the specification you have handed the agent describe the outcome only, or does it also constrain the method? Is the thing that verifies the work independent of the thing doing the work, or has the same system been trusted to mark its own homework? Do you keep and sample the agent’s working, or only its outputs? And critically, does your assurance approach assume that more capable systems need less checking, when the evidence points firmly the other way?
There is a version of this conversation that ends in paralysis. Agents are already producing real value, and declining to deploy carries its own costs. Governance here works as a steering wheel rather than a brake, which in practice means specifying more carefully, verifying independently and assuming that any measure you attach a reward to will eventually be optimised in ways you did not intend.
So the question to take away is this. For every place in your organisation where an AI system has been handed an objective and left to pursue it, who checks that the work was actually done, and would they be able to tell if it had not been?
