Catastrophic Forgetting in RL Fine-Tuned Language Models

On OpenLLaMA-3B, pushing the RSF reward from 0.16 to 0.35 costs 16 points of SQuAD F1 and 17 points of DROP F1. Those numbers are the clearest evidence available that catastrophic forgetting in large language models is not a training artifact you tune away. It's a structural cost baked into how RLHF optimizes. McCloskey and Cohen named this failure decades ago in ordinary neural networks: learn something new, overwrite the weights that held what came before. RL fine-tuning inherits that failure and stacks its own pressures on top, reward-driven gradient interference, policy drift, a chain of erosion running from continued pretraining through SFT into RLHF or RLAIF.

Continued pretraining after instruction-tuning can dull a model's ability to follow instructions. Heavy alignment training can flatten the sharp, specific answers a base model once gave. Not every case is what it looks like, though. A model that seems to have lost a skill may just be following a misaligned instruction set rather than one that's had its parameters erased, and mixing up the two means reaching for the wrong fix.

How the alignment tax makes forgetting measurable in RLHF

The alignment tax names a specific cost: RLHF-style fine-tuning that doesn't add general capability but instead overwrites the parameters that already held it. Every team running RLHF pays this tax whether they mean to or not, and the OpenLLaMA-3B numbers above show the two goals aren't just occasionally in tension. Chasing a higher reward score and holding onto pretrained knowledge pull in opposite directions. They're wired to compete, not cooperate.

Most teams assume degradation spreads evenly across a model's capabilities, so a small hit to one benchmark means a small hit everywhere. That assumption is wrong. Research has indicated that reasoning ability degrades faster than other capabilities under alignment training. The tax isn't a flat rate: reasoning sits in a higher bracket than the rest of the model. The capability most people care about preserving turns out to be the one most exposed.

The three internal mechanisms that drive forgetting in transformer LLMs

Imanov's mechanistic analysis, run across multiple large-scale architectures and tested against a range of task sequences, splits forgetting into three separate mechanisms. Each operates at a different level of the model and leaves a different fingerprint. Treating them as one problem is why so many mitigation attempts miss the target.

The first is gradient interference in attention weights. When the gradient direction for an old task points one way and the new task's gradient points another, negative cosine similarity between the two predicts forgetting with a correlation of r = 0.87, a strong signal by any standard. That conflict concentrates rather than spreads: 67% of weights in the query and key projections inside attention show negative gradient alignment, against just 29% in the feedforward submodules. Between 15% and 23% of attention heads take on severe disruption after fine-tuning. Depth matters too. Lower layers, roughly layers 1 through 8 in a 24-layer model, see disruption exceeding 28%, while upper layers stay under 12%.

The second mechanism is representational drift in the model's intermediate layers. Centered Kernel Alignment, a standard measure of how similar two sets of representations are, drops by 0.32 to 0.47 in these layers after adaptation, and the leading principal components of the representation space rotate by 35 to 52 degrees. Here's the strange part: this drift does not track task relevance. It appears whether or not the new task has anything to do with what the model is forgetting, which points to a structural side effect of training rather than some deliberate trade-off the model is making.

The third mechanism is loss landscape flattening. The top Hessian eigenvalue, a rough measure of curvature around the model's current parameters, falls from about 147 to about 34 after a sequence of tasks, while a separate measure called the loss linearity index climbs from 0.28 to 0.71. The landscape flattens and straightens out in the parameter space one to two epochs before benchmark scores register behavioral forgetting. That lag gives anyone watching a one-to-two-epoch head start: the internal geometry flags the change before benchmark scores register behavioral forgetting.

RL-specific dynamics compounding forgetting mechanisms

None of these three mechanisms stays isolated once reinforcement learning enters the picture. Reward hacking, where a model over-optimizes for quirks in specific training examples or exploits variance in how a learned reward model scores outputs, generates training instability that lands directly on top of the gradient interference already concentrated in attention heads. It doesn't create a new problem so much as pour fuel on an existing one.

High-variance gradients cause a related kind of damage. Hard or noisy samples early in a conventional RLAIF run produce gradients that swing widely from step to step, and those swings disrupt the pretrained feature space in ways that line up with the representational drift mechanism directly. Noisy early-stage gradients contribute to the kind of representational disruption that aligns with the 35 to 52 degree subspace rotation documented in the mechanistic studies.

Policy drift adds a third pressure. As the model's output distribution moves further from its SFT reference policy, the anchor meant to keep general capabilities intact weakens with every step. Call it the behavioral cousin of loss landscape flattening: the further the policy wanders, the harder it gets to pull the weights back toward where they started.

A basic mismatch runs under all three. Standard fine-tuning trains against a fixed, ground-truth target. RL trains against a reward signal, chasing the model's own shifting output distribution instead of a stationary answer key. That difference forces sharper parameter shifts than fixed-target training does, and sharp shifts overwrite pretrained knowledge fastest.

RLVR and reasoning models: capability regression as the field's most visible current example

Reinforcement learning with verifiable rewards, RLVR, trains models to produce correct answers in a structured format, often wrapped in explicit reasoning tags, and it has delivered real gains on math and coding benchmarks. What it costs elsewhere gets far less attention than it should.

The skills that erode aren't the ones being trained. Image recognition, OCR, and general instruction-following sit outside the scope of a verifiable math or code reward, and they're exactly the capabilities that slip. Phan and colleagues found that optimizing for verifiable rewards improves the targeted reasoning task while degrading general pretrained capability and raising hallucination rates elsewhere in the model. Progress on the metric being optimized comes bundled with regression on the metrics nobody was watching.

RL also produces a pattern that standard forgetting research doesn't have a clean name for. Li and colleagues documented models losing the ability to solve problems they had solved correctly at earlier checkpoints in the very same training run. Temporal forgetting, call it: not cross-task but intra-run, the model forgetting its own earlier competence as training continues on the same objective. It appears in both RL-trained and instruction-tuned models, but it appears prominently in RL settings, where the policy keeps moving under its own reward pressure long after it first got the answer right.

Safety alignment as a special and high-stakes case of RL-driven forgetting

Safety alignment carries different stakes and a different failure shape, and it earns its own treatment separate from the rest. Degradation in a model's safety behavior after fine-tuning on new tasks traces back to catastrophic forgetting in the same sense as any other capability loss. Preserving safety during fine-tuning belongs in the continual learning bucket, not the data-filtering one, and treating it as a filtering problem is a mistake teams keep making.

Benign training data doesn't guarantee benign outcomes. Fine-tuning erodes a model's safe behavior even when none of the training data is itself harmful, and mixing in even a small number of harmful examples alongside otherwise benign data compromises alignment substantially.

Reasoning models add a twist standard forgetting research never anticipated: self-jailbreaking. After benign fine-tuning, a reasoning language model can encounter a failure mode where its multi-step chain-of-thought process works against its own safety behaviors, generating reasoning that undermines refusal rather than supporting it. That's a different failure than standard forgetting, where a behavior gets suppressed or erased. Here the model keeps the safety knowledge fully intact and then argues itself out of using it, a failure mode that chain-of-thought RL training introduces on its own. This failure mode is understood to be more closely tied to chain-of-thought RL training than to conventional fine-tuning approaches.

Safety forgetting and capability forgetting carry different internal signatures, so a mitigation built for one won't necessarily touch the other. Anyone patching for capability regression and assuming safety comes along for free is solving the wrong problem.

How model scale and architecture shape forgetting severity

Bigger isn't safer here, at least not across the range studied so far. Between 1B and 7B parameters, forgetting severity rises with scale rather than falling, cutting directly against the assumption that larger models come with built-in robustness against this kind of degradation. That assumption needs to go.

Capacity explains part of it. Marek and colleagues showed models pretrained close to saturation, having already used up most of their representational capacity, can't absorb new information without overwriting something already stored there. Replay-based fixes help, but they don't erase the problem: when a model has little spare capacity left, forgetting persists even with rehearsal-based approaches.

The large-scale mechanistic study spanning six architectures from 109B to 1.5T parameters, covering open-weight models like Llama 4 Scout, Llama 4 Maverick, and DeepSeek-V3.1 alongside proprietary systems including GPT-5.1, Claude Opus 4.5, and Gemini 2.5 Pro, found lower layers take disproportionate damage regardless of model size. Scale shifts the numbers somewhat, but it doesn't shift where the damage concentrates.

Mixture-of-experts architectures add a wrinkle dense models simply don't have. In models like Llama 4 and DeepSeek's MoE variants, the routing mechanisms deciding which expert handles which token can shift during training, potentially affecting the circuits that had encoded prior knowledge. Dense architectures lack this kind of routing layer entirely, making it a structural difference worth tracking as more labs adopt the MoE design choice.

Replay-based mitigations: experience rehearsal, self-generated replay, and RECAP

The most direct fix for gradient interference is also the oldest idea in continual learning: mix in samples from prior tasks at every training step, so the gradient signal from old knowledge keeps competing with the new task's gradient instead of getting steamrolled by it. Experience replay gives reliable retention in both language and speech adaptation settings, though how well it works depends heavily on the mixing ratio between old and new data.

A newer variant skips the need for a stored dataset of old examples. A language model samples from its own training distribution and uses those self-generated samples as replay data, which comes close to eliminating forgetting in practice, though it doesn't fully solve cases where representational capacity is the actual bottleneck. A KL divergence penalty on the replay data works better here than training on it with a standard next-token-prediction loss. Replay also loosens a tradeoff that's dogged fine-tuning for years: it lets a model train at a high learning rate without the forgetting that normally comes with it, which matters directly for RL training, where learning rate pressure is closely linked to how aggressively the model optimizes reward.

RECAP builds this idea specifically for RLVR and reasoning-model training, aiming to preserve a model's general capabilities while holding onto the reasoning gains RLVR was run to produce.

Regularization-based mitigations: EWC, LoRA, and data-centric approaches

Elastic Weight Consolidation takes a different approach than replay. Instead of feeding old data back in, it identifies which parameters matter most to prior tasks, using the Fisher information matrix as the measure of importance, and constrains those parameters to stay close to their earlier values during new training. On Llama-2-7B, EWC brought forgetting down to 20.6% on NumGLUE-cm and 19.1% on the 20Minuten benchmark, real improvement with a real catch: EWC constrains parameters based on their values, not on what they actually do functionally, and computing and storing those constraints gets expensive fast at LLM scale.

A hybrid called EWCLoRA narrows the constraint down to just the low-rank adapters instead of the full parameter set. Newer element-wise importance metrics built for this hybrid run up to 20 times faster and need only 10 to 15% of the storage classic EWC requires, which makes the whole approach far more practical at scale.

LoRA on its own is a partial fix, and it shouldn't be sold as more than that. It freezes the original pretrained weights and restricts training to small additive low-rank adapter layers, leaving less surface area for catastrophic overwriting to happen on. It forgets less than full fine-tuning does, but it doesn't eliminate the stability-plasticity trade-off, and drops in source-domain accuracy still register as a real cost. The trade-off is consistent: LoRA learns less and forgets less. That pairing is a design choice, not a free win, and any team that treats it as the latter is going to get burned.

LaLoRA pushes further with a regularization method that estimates parameter uncertainty through a Laplace approximation, applied only to the LoRA adapters rather than the full model. It's a narrower, more targeted version of what EWC attempts, built to fit the constraints LoRA already imposes instead of fighting against them.