Process Reward Models for Multi-Step Reasoning
Step-level reward models catch flawed reasoning that correct final answers hide.
Devon Achterberg
The Reward Signal — Page 2
Reward Model Engineering
Reward Model Checkpoint Selection Criteria
Picking your best checkpoint means watching for reward hacking, not just picking the highest score.
Pointwise vs Pairwise vs Listwise Reward Training Objectives
Choosing the right training objective shapes how well reward models learn from human feedback.
Cassiel Oduya
More From The Reward Signal
Length Bias in Reward Model Training
Small biases in human feedback compound into verbose AI through reinforcement learning loops.
Anjali Raghunathan
Reward Model Generalization Across Out-of-Distribution Prompts
Reward models collapse on out-of-distribution tasks, leaving alignment systems vulnerable to gaming.
Devon Achterberg
Verifiable Reward Design for Code Generation Tasks
Code's executability enables richer reward signals than human judgment can provide.
Marta Wierzbicka
Reward Model Ensembling for Uncertainty Estimation
Using multiple reward models to quantify and act on training uncertainty.
Cassiel Oduya
Preference Dataset Quality Metrics for RLHF
Small, high-quality preference datasets outperform larger, noisier ones in training reward models.
Anjali Raghunathan
Implicit Reward Models in Offline Preference Learning
DPO works on paper but falters on cyclic preferences and unbounded reward scaling.
Cassiel Oduya
Constitutional AI Self-Critique Loop Mechanics
A self-correcting loop proves less important than the principles guiding it.
Devon Achterberg
Direct Preference Optimization vs PPO for Instruction Tuning
DPO trains faster and simpler, but PPO explores better on hard tasks.
Marta Wierzbicka
KL Divergence Penalty Tuning in PPO for LLMs
Setting this hyperparameter wrong dooms alignment to either inertia or reward hacking.
Cassiel Oduya