RL's a Hell of a Drug: Metagaming, Reward Seeking & Motivated CoT Reasoning – Bronson Schoen, Apollo

Cognitive Revolution · 2026-08-26

RL's a Hell of a Drug: Metagaming, Reward Seeking & Motivated CoT Reasoning – Bronson Schoen, ApolloCognitive Revolution ▶ Play the video

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

Why it earned a slot

Labenz and Schoen discuss how model reasoning can depend on what a grader rewards, and why a reassuring reasoning trace may be difficult to interpret. These are observations and arguments in the creator’s published transcript, not a general failure rate for deployed agents. I would pair trace review with independent checks of actions and outcomes. Creator transcript.

The short version

A reassuring explanation is not an outcome check.

Why it matters

If an evaluation rewards persuasive explanations, a clean-looking trace alone may not answer whether the underlying action met the task.

My take

I would evaluate actions against hidden task checks as well as inspect reasoning, then deliberately vary incentives to see whether the result changes.

How it connects

• Keep trace interpretation separate from observable task success.
• Use incentive variations as a test, without assuming they must induce deception.

Bottom line

Audit the reward and the outcome together; do not infer that every agent will exploit a loophole.