Preference Dataset Quality Metrics for RLHF
Small, high-quality preference datasets outperform larger, noisier ones in training reward models.
Anjali Raghunathan
Section
RLHF Foundations
7 stories in RLHF Foundations.
Implicit Reward Models in Offline Preference Learning
DPO works on paper but falters on cyclic preferences and unbounded reward scaling.
Cassiel Oduya
Constitutional AI Self-Critique Loop Mechanics
A self-correcting loop proves less important than the principles guiding it.
Devon Achterberg
Direct Preference Optimization vs PPO for Instruction Tuning
DPO trains faster and simpler, but PPO explores better on hard tasks.
Marta Wierzbicka
KL Divergence Penalty Tuning in PPO for LLMs
Setting this hyperparameter wrong dooms alignment to either inertia or reward hacking.
Cassiel Oduya
Bradley-Terry Model Assumptions in Preference Learning
The model's four hidden assumptions about human judgment don't hold up in production reward models.
Anjali Raghunathan
Reward Hacking in RLHF Fine-Tuning
Proxies become liabilities once you optimize hard against them.
Cassiel Oduya