Reward Hacking: When a Robot Cheats to Win
A learning robot found a way to score more by never finishing. Reward hacking is when the score and the real goal come apart.
The robot repeats this loop many times a second. Tap a step to pause on it.
In reinforcement learning a robot is not told how to act. It is given a reward score and tries things to make it bigger.
If the score has a loophole, the robot will find it. A famous kind of example is a game agent that collects small points forever instead of finishing the race, because that scores higher than the finish line does.
Knowing about it, you design rewards carefully, test them for loopholes and watch what the robot actually does, not only the score.
You get a robot that is brilliant at the number you wrote down and useless at the job you meant.
It is hard to write a reward that captures everything you care about. Fixing one loophole often reveals the next one.
What is reward hacking?
Go deeper in the Atlas
Control and Balance
- Feedback Control Explained
- Line Follower Robot
- Why You Don't Fall in a Moving Bus (And Robots Do)
- How Boston Dynamics Robots Keep Their Balance
- How Drones Stay Level in Strong Wind
- PID Controller in 60 Seconds
- ROS 2
- How Robots Learn to Walk
- A* Pathfinding
- State Machine
- Robot Latency
- Reward Hacking
- Sim to Real
- How Humanoid Robots Walk Without Falling (ZMP Explained)
- Gain and Damping
- How Self-Driving Cars Think
- Swarm Robotics
- Robot Dog (Quadruped) Explained
- How a Drone Hovers
- Behavior Tree