Fast and Stable Initialization: Navigating LoRA Weights SVD Catalyzed Off-Policiness in RL

Publish

Why SVD-informed LoRA initializations that help SFT can destabilize RL, and what safer initialization geometry looks like.

LoRA in SFT vs RL: Why Initialization Becomes the Bottleneck

LoRA has become the standard method for parameter-efficient fine-tuning in the supervised fine-tuning (SFT) regime. Under SFT, the training objective is cross-entropy against fixed target sequences, which gives optimization two structural advantages.

First, the gradient signal is dense and has relatively low variance. Every token in every training example contributes a supervised gradient against a deterministic target, with no sampling noise or reward estimation. Second, SFT places no explicit restriction on how far the parameters may move from the pretrained model. How far the weights actually move depends on the task: fine-tuning on a domain close to pretraining data (e.g., general instruction following) may require only small adjustments, while adapting to a specialized domain (e.g., medical text, mathematical reasoning, or code generation) can require substantial weight changes. Together, these properties provide the optimizer with a large effective budget of reliable gradient steps and enough freedom to exploit structured initializations without being destabilized by them. In practice, better initializations of the low-rank factors often accelerate convergence and can improve final performance, precisely because the optimizer has sufficient room to capitalize on a favorable starting point. As a result, SFT is compatible with a wide range of initialization strategies, including those that induce relatively large initial weight perturbations.

Reinforcement learning with verifiable rewards (RLVR) operates under fundamentally different dynamics, and these dynamics turn initialization from a lever for improvement into a potential point of failure. The first-order approximation makes RL training fragile: The token-level objectives used by standard RL algorithms for LLMs (REINFORCE, GRPO) are not exact optimizers of the true sequence-level reward. They are first-order approximations, and their validity depends on a condition that initialization can violate.

The central issue is that the token-level objectives used in standard LLM RL algorithms, such as REINFORCE and GRPO, are not exact optimizers of the true sequence-level reward. Rather, they are first-order approximations whose validity depends on the updated policy remaining sufficiently close to the rollout policy. Initialization, therefore, matters not only through optimization geometry, but also through whether this approximation remains trustworthy at all.

The true objective in RLVR is the expected sequence-level reward: J(θ)=Ex∼D, y∼πθ(⋅∣x)[R(x,y)]J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)], where πθ\pi_\theta is the current policy and R(x,y)∈{0,1}R(x,y) \in \{0,1\} is a verifiable reward assigned to the complete response yy. Since the sequence probability decomposes as πθ(y∣x)=∏t=1Tπθ(yt∣x,y<t)\pi_\theta(y|x) = \prod_{t=1}^{T} \pi_\theta(y_t|x, y_{<t}), the REINFORCE gradient is:

∇θJ(θ)=Ey∼πθ ⁣[R(x,y)∑t=1T∇θlog⁡πθ(yt∣x,y<t)].\nabla_\theta J(\theta) = \mathbb{E}_{y \sim \pi_\theta}\!\left[R(x,y)\sum_{t=1}^{T} \nabla_\theta \log \pi_\theta(y_t|x, y_{<t})\right].

The sequence reward is naturally distributed across all tokens: the quality of a complete response affects the gradient at every token position.

In practice, responses are not sampled from the current policy πθ\pi_\theta in the training engine but from a rollout policy μ\mu in the inference engine. This introduces an importance sampling (IS) correction:

∇θJ(θ)=Ey∼μ[R(x,y)πθ(y∣x)μ(y∣x)∑t=1T∇θlog⁡πθ(yt∣x,y<t)].\nabla_\theta J(\theta) = \mathbb{E}_{y \sim \mu}\left[R(x,y)\frac{\pi_\theta(y|x)}{\mu(y|x)}\sum_{t=1}^{T} \nabla_\theta \log \pi_\theta(y_t|x, y_{<t})\right].

The sequence-level IS weight decomposes into a product of per-token ratios:

πθ(y∣x)μ(y∣x)=∏t=1Tπθ(yt∣x,y<t)μ(yt∣x,y<t)=∏t=1Trt=∏t=1T(1+δt),\frac{\pi_\theta(y|x)}{\mu(y|x)} = \prod_{t=1}^{T} \frac{\pi_\theta(y_t|x, y_{<t})}{\mu(y_t|x, y_{<t})} = \prod_{t=1}^{T} r_t = \prod_{t=1}^{T} (1 + \delta_t),

where δt=rt−1\delta_t = r_t - 1 measures the per-token deviation between the current and rollout policies. This product is exponentially unstable: for a sequence of length T=512T = 512, even a modest per-token ratio of rt=1.01r_t = 1.01 yields a sequence-level weight of 1.01512≈1631.01^{512} \approx 163. This is why direct sequence-level IS produces extreme variance in LLM RL training and is rarely used in practice.

To avoid the exploding product, Zheng et al. (2025) observe that when πθ\pi_\theta and μ\mu are close, i.e., δt\delta_t is small for all tt, the product admits a first-order Taylor expansion:

∏t=1T(1+δt)≈1+∑t=1Tδt+O(δ2).\prod_{t=1}^{T}(1 + \delta_t) \approx 1 + \sum_{t=1}^{T}\delta_t + O(\delta^2).

Dropping the higher-order terms and substituting back, the sequence-level gradient is approximated by a token-level surrogate:

∇θJ(θ)≈Ey∼μ[R(x,y)∑t=1Twt∇θlog⁡πθ(yt∣x,y<t)],\nabla_\theta J(\theta) \approx \mathbb{E}_{y \sim \mu}\left[R(x,y)\sum_{t=1}^{T} w_t \nabla_\theta \log \pi_\theta(y_t|x, y_{<t})\right],

where wt=πθ(yt∣x,y<t)μ(yt∣x,y<t)w_t = \frac{\pi_\theta(y_t|x, y_{<t})}{\mu(y_t|x, y_{<t})} is the per-token IS ratio. This is precisely the objective that REINFORCE and GRPO optimize in practice: a sum over tokens, each weighted by its own token-level ratio wtw_t, rather than the intractable sequence-level product. The mathematical foundation of most LLM RL methods is this first-order surrogate.

The requirement that rt≈1r_t \approx 1 is not merely a mathematical convenience. It is the regime in which the entire surrogate objective remains meaningful. Zhu et al. (2025) formalize this result similarly as Gate I (KL Anchor) of their Three-Gate Theory. To second order, the KL divergence between the updated and current policies is governed by the Fisher information matrix: DKL(πθ+Δθ∣πθ)≈12Δθ⊤FΔθ,D_{\text{KL}}(\pi_{\theta + \Delta\theta} | \pi_\theta) \approx \frac{1}{2}\Delta\theta^\top F\Delta\theta, where F=E[∇θlog⁡πθ⊤∇θlog⁡πθ]F = \mathbb{E}[\nabla_\theta \log \pi_\theta^\top \nabla_\theta \log \pi_\theta] is the Fisher information matrix. The KL penalty, importance sampling clipping, and trust region methods (TRPO, PPO) all serve the same purpose: they keep the policy update small enough that the first-order approximation remains valid.

When a single-step update satisfies DKL(πθ+∣πθ)≤KD_{\text{KL}}(\pi_{\theta^+} | \pi_\theta) \leq K and the Fisher information is bounded below by F(θ)⪰mIF(\theta) \succeq m I on the update subspace, the weight change in any block W⊂θW \subset \theta is bounded: ∣ΔW∣F≤2Km(1+o(1)).|\Delta W|_F \leq \sqrt{\frac{2K}{m}}(1 + o(1)). This is the KL leash: the KL penalty in the RLVR objective imposes an implicit ceiling on the magnitude of weight updates at every training step. Unlike SFT, where the optimizer can move as far as the loss surface demands, RLVR confines the policy to a narrow neighborhood of the pretrained weights throughout training. Stable Training, therefore, has to make progress through small, controlled steps rather than big jumps.

Initialization does not just decide where optimization starts. It also shapes the direction the whole training trajectory tends to follow. In RLVR, that matters a lot. The optimizer is only allowed to move inside a small local neighborhood, but not all directions inside that neighborhood are equally safe. Some are much more aggressive: they push the policy outward faster and make it easier to leave the region where the first-order approximation is reliable. Therefore, the problem with a bad initialization is not simply that its first step is too large. The deeper issue is that it builds in an inductive bias toward a dangerous optimization path, making it much more likely that training will drift outside the valid local regime and become unstable.

The question in RLVR is not only "which initialization reaches the best solution?" but also "which initialization keeps the first-order approximation valid enough for the optimizer to do its job?"

Geometry prior takes the path

Recent RLVR studies have observed that SVD-informed LoRA methods designed for SFT such as PiSSA and MiLoRA can become unstable or even collapse during RL training. The common pattern is that spectral initialization makes RL optimization too aggressive too early.

To make this precise, let the pretrained weight matrix be W0∈Rm×n,W_0 \in \mathbb{R}^{m \times n}, with singular value decomposition W0=UΣV⊤,W_0 = U \Sigma V^\top, where Σ=diag⁡(σ1,σ2,…,σk),σ1≥σ2≥⋯≥σk≥0.\Sigma = \operatorname{diag}(\sigma_1,\sigma_2,\dots,\sigma_k), \qquad \sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_k \ge 0. We write the trainable layer as W=Wres+BA,W = W_{\mathrm{res}} + BA, where WresW_{\mathrm{res}} is frozen, and BABA is the trainable low-rank part.

PiSSA uses the principal singular subspace of W0W_0. Let Wtop=UrΣrVr⊤W_{\mathrm{top}} = U_r \Sigma_r V_r^\top denote the rank- rr reconstruction from the top singular components. PiSSA freezes Wres=W0−Wtop,W_{\mathrm{res}} = W_0 - W_{\mathrm{top}}, and initializes the trainable low rank adapters B0=UrΣr1/2,A0=Σr1/2Vr⊤,B_0 = U_r \Sigma_r^{1/2}, A_0 = \Sigma_r^{1/2} V_r^\top, which gives B0A0=UrΣrVr⊤=Wtop.B_0A_0 = U_r \Sigma_r V_r^\top = W_{\mathrm{top}}. MiLoRA instead uses the minor singular subspace. Let Wtail=U−rΣ−rV−r⊤W_{\mathrm{tail}} = U_{-r} \Sigma_{-r} V_{-r}^\top denote the rank- rr reconstruction from the tail singular components. PiSSA freezes Wres=W0−Wtail,W_{\mathrm{res}} = W_0 - W_{\mathrm{tail}}, and initializes the trainable low rank adapters B0=UtailΣtail1/2,A0=Σtail1/2Vtail⊤,B_0 = U_{\mathrm{tail}} \Sigma_{\mathrm{tail}}^{1/2}, A_0 = \Sigma_{\mathrm{tail}}^{1/2} V_{\mathrm{tail}}^\top, so that B0A0=Wtail.B_0A_0 = W_{\mathrm{tail}}. By construction, PiSSA and MiLoRA bias the trainable update toward a specific spectral subspace of the pretrained weight matrix. This gives the optimizer a strong geometric preference from the first step, so the induced gradient is often larger and more concentrated than in a neutral low-rank parameterization. Early training reward may improve quickly, but the policy also drifts away from the rollout policy much faster, causing KL to rise more rapidly. RL training depends on a local first-order approximation, which is only reliable when this policy shift remains small. Once the KL drift exceeds that local region, the approximation breaks down: the token-level surrogate no longer reflects the true sequence-level improvement direction, and training can suddenly deteriorate into instability or collapse.

Therefore, the collapse mechanism is spectral inductive bias  →  faster early updates  →  larger gradients and KL drift  →  breakdown of first-order approximation  →  collapse.\text{spectral inductive bias} \;\rightarrow\; \text{faster early updates} \;\rightarrow\; \text{larger gradients and KL drift} \;\rightarrow\; \text{breakdown of first-order approximation} \;\rightarrow\; \text{collapse}.

Learning Rate Decay Mitigates Spectral Risk

The previous section already tells us what the potential fix could be. If geometry-aware initialization leads to larger gradient norms and faster KL growth, then the optimizer is simply moving too aggressively during RL training. A natural way to calm this down is to anneal the learning rate.

This picture makes the effect of learning-rate annealing very concrete. In both cases, replacing a constant learning rate with cosine decay delays and mitigates the late-stage collapse, which is exactly what we would expect if instability is caused by overly aggressive updates.

Once the learning rate is annealed, the optimizer moves more cautiously, the gradient norm and KL grow more slowly, and the trajectory stays longer inside the local regime where RL remains stable.

We observe that the two initializations respond very differently. For MiLoRA, cosine decay is sufficient to largely recover normal training behavior: the reward curve remains much more stable, and the severe downward drift seen under a constant learning rate is substantially reduced. This suggests that MiLoRA's geometry is risky but still manageable: once the update magnitude is controlled, the optimizer can remain on a workable path.

PiSSA is different. Although cosine decay also slows the collapse, it does not fundamentally rescue training: performance still deteriorates sharply and settles into a bad regime. This tells us that PiSSA's problem is not just that the learning rate is too large. The deeper issue is that its initialization biases RL toward a more dangerous direction from the start. Even after we reduce the step size, the trajectory is still off the safe path.

This is exactly the pattern predicted by the view that RLVR learns off the principal directions of the pretrained model (Zhu et al., 2025). Their argument is that, under KL-constrained updates, RL optimization is naturally steered away from principal directions and toward lower-curvature, more spectrum-preserving subspaces. From this perspective, principal-targeted initialization is risky not simply because it can make updates larger, but because it pushes training toward directions that RL itself does not naturally favor. Learning-rate decay can soften this aggressiveness, but it cannot fully remove the directional bias introduced at initialization. That is why MiLoRA can often be stabilized by annealing the learning rate, while PiSSA remains much harder to rescue: once RL is pushed too strongly toward the principal subspace, the trajectory is already closer to the dangerous side of the geometry (Zhu et al., 2025).

The geometric intuition is straightforward. The safe local region around the rollout policy is better thought of as an ellipse than a sphere, meaning that equal-sized parameter updates can have very different consequences depending on their direction. Along some directions, the optimizer can move relatively far while keeping policy drift and KL growth under control. Along with others, even a modest step can produce a much sharper departure from the rollout policy. Therefore, the real issue is not only update magnitude, but spectral risk: some directions are far less forgiving than others.

This is why lowering the learning rate helps. It does not change the direction favored by initialization, but it reduces how aggressively optimization moves along that direction. If the chosen direction is risky, this extra caution can keep training inside the safe local region for much longer, which makes collapse much less likely.

From Spectral Geometry to Orthonormal Geometry

The discussion above does not mean that geometry is the wrong idea for RL. The real lesson is more subtle: RL prefers gentler geometry. From this perspective, orthonormal geometry is a much more natural candidate.

Why? Because an orthonormal basis gives structure without amplification. Its directions are balanced, mutually orthogonal, and well-conditioned. Unlike spectral initialization, it does not privilege some directions simply because the pretrained model assigns them large singular values. In practical terms, this makes the low-rank parameterization less biased toward sharp, high-gain updates and more compatible with the local, trust-region style of optimization that RL prefers. This intuition is also consistent with prior LoRA literature showing that orthonormal initialization can accelerate convergence and improve training behavior compared with standard LoRA-style initialization, even outside RL.

This is the perspective from which we introduce Wavelet-based orthonormal initialization. H∈Rn×nH \in \mathbb{R}^{n \times n} be an orthonormal wavelet basis matrix, and initialize LoRA as B0=0m×r,A0=H1:r,:,B_0 = 0_{m \times r}, A_0 = H_{1:r,:}, where H1:r,:H_{1:r,:} denotes the first rr rows of the wavelet basis. By construction, A0A0⊤=Ir.A_0 A_0^\top = I_r. Wavelet initialization is still geometric, but it is a model-agnostic geometry rather than a pretrained-spectrum geometry. That difference matters. It keeps the conditioning benefit of orthonormal structure, but avoids tying optimization too strongly to the singular directions of W0W_0.

It also has a very practical advantage: it does not require any SVD pre-computation. Methods such as PiSSA and MiLoRA depend on singular vectors of the pretrained weight matrix before training starts, which introduces extra preprocessing overhead. Wavelet initialization avoids this entirely while still giving a structured orthonormal basis. In practice, that makes it especially attractive when many layers must be initialized or when simplicity matters as much as final performance.

The left panel plots reward over training, and the right panel plots KL on a log scale. Compared with vanilla LoRA, Wavelet-LoRA not only remains competitive, but in fact achieves slightly higher reward and faster convergence. At the same time, its KL curve stays slightly lower and smoother in the later stage, suggesting that the optimization path is somewhat gentler and less prone to the kind of aggressive drift that often destabilizes RL.

The downstream evaluation results further support this picture: Wavelet-LoRA is not only stable during training, but also improves over standard LoRA on overall reasoning tasks.

MethodGSM8KMATH500AIME22AIME23AIME24AIME25Average
LoRA0.7539 ± 0.00570.7127 ± 0.00570.4333 ± 0.05770.4222 ± 0.01920.7222 ± 0.01920.3333 ± 0.00000.5629
Wavelet-LoRA0.7528 ± 0.00200.7013 ± 0.01140.5000 ± 0.03330.4222 ± 0.05090.7556 ± 0.01920.3667 ± 0.03330.5831

For RL fine-tuning, the best initialization is not the one that moves fastest at the beginning, but the one that stays stable long enough to keep improving.

References

Author

Mind Lab

Core Contributors

Ruijia Zhang, Kieran Liu, Di Zhang, Andrew Chen, Pony Ma

Team

Andrew Chen, Kaijie Chen, Song Cao, Cleon Cheng, Steven Chiang, Kaixuan Fan, Huan Feng, Nolan Ho, Charles Huang, Fancy Kong, Kyrie Lei, Andrew Lei, Alyssa Li, Lucian Li, Ray Li, Theo Li, Logan Liu, Kieran Liu, Xiang Liu, Irvine Lu, Runze Lv, Pony Ma, Vincent Wang, Rio Yang, Shiro Yang, Maxwell Yao, Regis Ye, Salmon Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Sueky Zhang, Adrian Zhou, Xinyue Zhu, Murphy Zhuang and Mindverse Team

Names are listed alphabetically within team.

Citation

@misc{ruijiazhang2026fastandstableinit,
  author = {Ruijia Zhang and Kieran Liu and Di Zhang and Andrew Chen and Pony Ma and {Mind Lab}},
  title = {Fast and Stable Initialization: Navigating LoRA Weights SVD Catalyzed Off-Policiness in RL},
  year = {2026},
  howpublished = {Mind Lab: A Lab for Experiential Intelligence},
  note = {https://macaron.im/mindlab/research/fast-and-stable-initialization-navigating-lora-weights-svd-catalyzed-off-policiness-in-rl}
}