AI Safety · Human-AI Collaboration
Two Stanford researchers just published a blueprint for a question the whole AI industry keeps dodging: as AI acts more on its own, who actually stays in charge?
Their answer isn't a rulebook. It's a game — one where the AI learns exactly when to ask, and the human learns exactly when to answer.
Stanford Graduate School of Business researchers William Overman and Mohsen Bayati have published two papers tackling how humans stay meaningfully in control as AI systems act more autonomously. The first, "The Oversight Game," models the human-AI relationship as a cooperative game with no winner or loser, testing it in a grid environment called Lavaland where an learns on its own when to pause and ask for human guidance near hidden danger. The second paper, on calibrated collective oversight, tackles a messier real-world case: overseeing a powerful AI system you didn't build and don't fully trust, by pooling several weaker overseers into a system with a mathematically guaranteed safety rate. Both papers are available on the arXiv preprint server, and the researchers see them as building blocks for the wider effort to solve AI alignment.
The core question behind both papers: as AI agents act more independently, how do humans stay meaningfully in control?
Two New Papers Just Mapped Out How Humans Stay in Control of AI
William Overman started his Ph.D. at Stanford Graduate School of Business just two months before ChatGPT launched publicly in November 2022 — a genuinely unusual vantage point to watch a field explode from.
Even as he began using AI in his own research, Overman grew wary of where the technology was headed, worried less about dramatic worst-case scenarios and more about subtle, cumulative harm.
He teamed up with his adviser, Mohsen Bayati, a professor of operations, information, and technology at Stanford GSB, and turned to game theory, reinforcement learning, and causal inference to work the problem.
The result is two papers, both built around one central question: as AI systems act more on their own, how do humans stay meaningfully in control?
Bayati frames the split between the two papers directly: the team is developing frameworks to think about two classes of problems: collaboratively building a safe AI and controlling a potentially unsafe but very powerful one.
One paper designs a relationship from scratch. The other works with an AI system you didn't build and don't fully trust. Both are available on the arXiv preprint server.
Why "Just Add Guardrails" Isn't Actually a Plan
Overman's underlying concern isn't primarily about dramatic, headline-grabbing AI failures. It's about something quieter: misaligned AI overstepping its bounds without any malicious intent at all.
Small, subtle overreach, repeated across enough interactions, can add up to real harm — even when no single decision looks obviously wrong in isolation.
Overman puts the stakes in plain terms: getting this right matters for the future of human flourishing, not just for avoiding worst-case outcomes.
His view is that the interactions, training, and incentives shaping AI agents need to be deliberately designed now, rather than patched together after problems already show up.
That's a meaningfully different starting point than bolting on safety rules after the fact. It treats oversight as something to architect, not something to retrofit.
A Game With No Winner or Loser
The first paper, titled "The Oversight Game," makes a deliberate framing choice: it models the human-AI relationship as a game, but explicitly one without a winner or a loser.
Overman describes the spirit of it directly: it's more like cooperative party games, where you're trying to accomplish something together, rather than a contest either side is trying to win.
Self-driving cars offer a useful real-world parallel. Each car pursues its own destination, but every vehicle on the road shares an overriding interest in avoiding collisions with the others.
In the researchers' game, an AI agent can choose, at any moment, to act autonomously or defer to a human who can override or shut it down entirely.
The human, simultaneously, chooses whether to trust the AI's judgment or step in — with intervention only possible when the AI has actually deferred, and the human has chosen to oversee.
The overarching goal is training the AI to recognize the right moments to pause and ask, rather than either never asking or asking constantly.
Learning to Avoid Lava, Without Being Told How
To test the idea, the researchers built an environment called Lavaland — a grid the AI agent has to cross to reach a goal, with patches of lava scattered along the way.
Critically, the AI agent was never explicitly trained to recognize the lava. Left entirely on its own, it would walk straight through it, triggering a steep penalty each time.
Over repeated attempts, without being told to coordinate, the AI agent and the human independently learned complementary behaviors: the agent learned to ask for guidance as it neared lava, and the human learned to step in with a safe route around it.
When there's no danger nearby, the pattern flips back automatically: the agent defaults to acting freely, and the human defaults to trusting its judgment.
This wasn't purely theoretical either. In tests with frontier models on agentic coding tasks, the AI independently learned to check in before risky actions, while the human learned when stepping back was safe.
What Happens When You Don't Trust the AI You're Using
The second paper tackles a messier, far more common real-world scenario: a company wants to use a powerful open-source AI model built by an entity it doesn't fully trust.
Retraining the model isn't an option, and redesigning the role it plays isn't realistic either — the model's raw performance is simply too valuable to walk away from.
The researchers' response to that constraint is straightforward to state: if you can't change the model, you watch it, carefully and systematically, instead.
Their method is called calibrated collective oversight, and it's built specifically for situations where no single overseer, human or AI, is strong enough to be trusted entirely on its own.
Calibrated Collective Oversight: Weak Watchers, Strong Guarantee
The core idea behind calibrated collective oversight is pooling several weaker, imperfect overseers, so their combined judgment can rein in a much stronger AI system none of them could individually control.
| The Oversight Game | Calibrated Collective Oversight |
|---|---|
| Designs a relationship from scratch | Works with an AI you already have, unmodified |
| One AI, one human, learning together over time | Multiple weak overseers pooled to watch one strong system |
| Needs a practice environment to train in | Needs only a set of overseers you already trust |
| Produces a stable, learned pattern of trust | Produces a mathematically guaranteed safety rate |
What sets this approach apart is a genuinely rare feature: the researchers can mathematically guarantee the rate of bad outcomes will stay below whatever threshold a user sets in advance.
Bayati puts the tunability plainly: you just tune the safety component to set the acceptable rate of unsafe decisions, whether that's 5 percent, 1 percent, or something else entirely.
The team tested this on a modified software-engineering benchmark, where weaker overseers successfully held an adversarially misaligned coding agent in check.
They also tested it on MACHIAVELLI, a game specifically built to measure how AI systems trade off completing a task against acting ethically. In both tests, violation rates tracked closely with the safety target dialed in.
Which Approach Is Actually Ready to Use Today
Of the two frameworks, Bayati sees calibrated collective oversight as the one closer to practical, real-world deployment right now.
Part of the appeal is genuine simplicity — all it requires is a set of weaker systems you already trust, whether those are human overseers or smaller AI models.
The Oversight Game, by contrast, needs practice before it's genuinely deployable. The AI and the human have to play it repeatedly in the actual setting where it'll be used — a doctor's office, a classroom, or somewhere else entirely.
Bayati points to a concrete example of why that repeated cost matters: an AI assistant meant to save a doctor time becomes counterproductive if it asks for oversight too often, burning exactly the time it was meant to save.
But acting autonomously too often carries its own cost too — more mistakes, made with less human awareness of what's happening in the moment.
Building well-designed practice environments for the Oversight Game is, in Bayati's own words, a significant undertaking — meaningful, but not yet a quick deployment.
Manohar Insight
What I find genuinely refreshing about this work is the refusal to treat "human in the loop" as a slogan rather than an engineering problem.
Most AI oversight conversations stop at the level of policy statements. This research goes further, building actual mathematical structures where both deferring and overseeing carry real costs — which is exactly what makes the resulting behavior trustworthy rather than performative.
The Lavaland result is the one I'd point skeptics toward first. Neither the AI nor the human was told to coordinate — the coordination emerged because the incentive structure made it the sensible thing to do.
That's a genuinely different kind of safety claim than "we added a safety layer." It's closer to "we designed the incentives so safety is the natural outcome."
The mathematical guarantee behind calibrated collective oversight is the more immediately useful contribution, precisely because it doesn't require trusting a single powerful system at all — only a handful of weaker ones you already do.
I'd treat both papers as exactly what the researchers call them: tools for others to build on, not finished products ready to ship into every AI deployment tomorrow.
The real contribution here isn't a specific algorithm. It's proof that "staying in control of AI" can be an engineering target, with guarantees, rather than just a hope.
Join the Signal
Between a system that learns trust through practice and one that offers a mathematical safety guarantee out of the box, which would you rather have overseeing a powerful AI system today? Share your take in the comments.
Discussion