Anjali Raghunathan

Faster reward scoring is worth nothing if it tanks the quality of training signals.

Decoupling rollout generation from training eliminates PPO's sequential bottleneck.

How to budget each RLHF stage separately before costs spiral.

Miscalibrated reward models exploit gaps between proxy scores and actual human preference.

Picking your best checkpoint means watching for reward hacking, not just picking the highest score.

Small biases in human feedback compound into verbose AI through reinforcement learning loops.

Small, high-quality preference datasets outperform larger, noisier ones in training reward models.

The model's four hidden assumptions about human judgment don't hold up in production reward models.