Reward Model Serving Latency in Online RL Training Loops
Faster reward scoring is worth nothing if it tanks the quality of training signals.
Anjali Raghunathan
The Reward Signal
RLHF Tooling
Experiment Tracking for RLHF Runs With Weights and Biases
Track RLHF experiments across three stages to catch failures before they compound downstream.
Annotation Platform Selection for Preference Data Collection
How to choose a platform that captures the actual quality of preference data.
Devon Achterberg
More From The Reward Signal
Distributed Rollout Generation for Large-Scale PPO Training
Decoupling rollout generation from training eliminates PPO's sequential bottleneck.
Anjali Raghunathan
TRL Library Architecture for RLHF Practitioners
Understanding TRL's stable and experimental trainer boundaries unlocks RLHF at scale.
Devon Achterberg
GRPO vs PPO for Large Language Model Fine-Tuning
GRPO eliminates the critic model, cutting memory costs without sacrificing performance.
Devon Achterberg
Catastrophic Forgetting in RL Fine-Tuned Language Models
Reward optimization in RL fine-tuning systematically erases previously learned capabilities.
Cassiel Oduya
Iterative Online RLHF With Continuously Updated Preference Data
Continuously updating preference data beats training reward models once on static data.
Marta Wierzbicka
Rejection Sampling Fine-Tuning as an RL Alternative
Rejection sampling matches reinforcement learning on math tasks without the complexity.
Devon Achterberg
Compute Budget Allocation Across RLHF Pipeline Stages
How to budget each RLHF stage separately before costs spiral.
Anjali Raghunathan
Multi-Stage Post-Training Order for Instruction-Following Models
The field is discovering that post-training stages must run in strict sequence, not interchangeably.
Devon Achterberg
Calibration of Learned Reward Models Against Human Ratings
Miscalibrated reward models exploit gaps between proxy scores and actual human preference.
Anjali Raghunathan