The Off-Switch Game

Research reconstruction

Illustrative numbers in a simplified version of the paper's model. Not a test of any real AI system. The human knows whether the robot's plan is good (+10) or bad (−10); the robot only has a belief.

Set up the game

Robot's confidence the action is good Chance that U is +10 rather than −10, in the robot's belief.

How often the human gets it wrong 0% is the paper's perfectly rational human, who presses the switch exactly when the action is bad.

Expected payoff of each move

Incentive to keep the off switch

Value of waiting minus the better of acting now or switching off. The paper calls this the robot's incentive to defer.

What the robot does here

The paper adds that this uncertainty isn't free. An agent that is overconfident about its objective will be difficult to correct, and one that is under-confident will be ineffective.

Assumptions and limits

It is a one-shot game with one human and one robot.
The robot treats the human's press of the switch as information about the objective, and the switch is its only source of that information.
The human is assumed to be approximately rational, and disabling the switch costs the robot nothing.
If the robot's model of the world or of the human is misspecified, the incentive to defer can disappear (Carey 2018, “Incorrigibility in the CIRL Framework”, AIES).
The result is a theorem about an abstract model, not a real-world safety guarantee. Today's language models are not built as explicit Bayesian agents like R.

Source: Hadfield-Menell, Dragan, Abbeel & Russell, “The Off-Switch Game”, IJCAI 2017; Carey 2018. The paper models human error with a noisily rational (Boltzmann) policy and a continuous belief; this figure uses a two-outcome belief and a flat error rate to keep the arithmetic visible.