% Source: https://kexue.fm/archives/11729
\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\usepackage{geometry}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{amsmath,amssymb}
\usepackage{booktabs,longtable,array}
\usepackage{xcolor}
\usepackage{listings}

\geometry{a4paper, margin=1in}
\definecolor{skyblue}{RGB}{0, 176, 240}
\newcommand{\mclip}[2]{\operatorname{mclip}(#1; #2)}
\newcommand{\msign}[1]{\operatorname{msign}(#1)}
\newcommand{\SVDone}{\operatorname{SVD1}}
\newcommand{\clip}[3]{\max(\min(#1, #2), #3)}

\title{Beyond MuP: 4. Preserving Parameter Stability}
\author{Su Jianlin}
\date{2026-04-24}

\begin{document}
\maketitle

Through derivations and calculations in previous articles, we can observe that the three stability metrics proposed in the first article \href{https://kexue.fm/archives/11340}{\emph{Beyond MuP: 1. Three Characteristics of a Good Model}} can generally be divided into \emph{parameter stability} and \emph{increment stability}. In \href{https://kexue.fm/archives/11605}{\emph{Beyond MuP: 2. Linear Layers and Steepest Descent}} and \href{https://kexue.fm/archives/11647}{\emph{Beyond MuP: 3. Special Cases Require Special Treatment}}, we demonstrated the process of integrating increment stability with steepest descent to obtain new update rules (optimizers).

However, for parameter stability, we previously only considered initialization. The task of this article is to explore how to maintain parameter stability throughout the entire training process, thus completing the practical application of the theory.

\section{Background}
\label{背景}

Taking \href{https://kexue.fm/archives/11605}{\emph{Beyond MuP: 2. Linear Layers and Steepest Descent}} as an example, the three stability metrics are:
\begin{align}
&\text{Forward Stability:}\quad\max_{\|\boldsymbol{x}\|_{RMS}=1} \| \boldsymbol{x}\boldsymbol{W}\|_{RMS} = \sqrt{\frac{d_{in}}{d_{out}}}\|\boldsymbol{W}\|_2 \\[5pt]
&\text{Dependency Stability:}\quad\max_{\|\boldsymbol{x}_1\|_{RMS}=\|\boldsymbol{x}_2\|_{RMS}=1} \| \boldsymbol{x}_1\boldsymbol{W} - \boldsymbol{x}_2\boldsymbol{W}\|_{RMS} = 2\sqrt{\frac{d_{in}}{d_{out}}}\|\boldsymbol{W}\|_2 \\[5pt]
&\text{Update Stability:}\quad\max_{\|\boldsymbol{x}\|_{RMS}=1} \| \boldsymbol{x}(\boldsymbol{W} + \Delta\boldsymbol{W}) - \boldsymbol{x}\boldsymbol{W}\|_{RMS} = \sqrt{\frac{d_{in}}{d_{out}}}\|\Delta\boldsymbol{W}\|_2
\end{align}
where $\boldsymbol{W}\in\mathbb{R}^{d_{in}\times d_{out}}$ is the parameter of a linear layer. We desire these three metrics to be $\Theta(1)$, which means we want the norm of the parameter and its increment to satisfy $\|\boldsymbol{W}\|_2 = \Theta(\sqrt{d_{out}/d_{in}})$ and $\|\Delta\boldsymbol{W}\|_2 = \Theta(\sqrt{d_{out}/d_{in}})$, respectively. In \href{https://kexue.fm/archives/11647}{\emph{Beyond MuP: 3. Special Cases Require Special Treatment}}, we performed similar calculations for layers such as Embedding and LM Head, with analogous conclusions, though the corresponding norms differ.

We used the increment condition as a stability metric and, based on the principle of steepest descent under ``stable acceleration,'' derived theoretically optimal update rules. For linear layers, this leads to the Muon optimizer:
\begin{equation}
\operatorname{argmin}_{\|\Delta\boldsymbol{W}\|_2\leq\eta\sqrt{\frac{d_{out}}{d_{in}}}} \operatorname{tr}(\boldsymbol{G}^{\top}\Delta\boldsymbol{W}) \qquad \Rightarrow \qquad \Delta\boldsymbol{W} = -\eta\sqrt{\frac{d_{out}}{d_{in}}}\msign(\boldsymbol{G})
\end{equation}
For parameter stability, we previously only required that initialization satisfies $\|\boldsymbol{W}\|_2 = \Theta(\sqrt{d_{out}/d_{in}})$. How to ensure such parameter stability is maintained throughout the entire training process remains unclear.

\section{Preliminary Thoughts}
\label{Preliminary Thoughts}

How can we ensure that $\boldsymbol{W}$ maintains $\|\boldsymbol{W}\|_2 = \Theta(\sqrt{d_{out}/d_{in}})$? More generally, given a parameter $\boldsymbol{\omega}$, which could be a vector, matrix, or even a higher-order tensor, a norm $\|\cdot\|$, typically induced by forward or dependency stability, and a scale $\tau$, how can we ensure that $\boldsymbol{\omega}$ maintains $\|\boldsymbol{\omega}\| = \Theta(\tau)$ during training?

A naive idea is to directly enforce $\|\boldsymbol{\omega}\| = \tau$ (where $\tau$ can be replaced by a constant multiple, but this does not affect the following discussion). The simplest implementation is to re-scale the norm to $\tau$ after each optimization step via normalization (e.g., \href{https://whenwen.github.io/wd_blog/public/hyperball-part-1.html}{Hyperball} and \href{https://papers.cool/arxiv/2511.18890}{Nemotron-Flash}). Another approach is to replace the original model parameterization with normalized ones, i.e., change $\boldsymbol{f}(\boldsymbol{x};\boldsymbol{\omega})$ to $\boldsymbol{f}(\boldsymbol{x};\tau\boldsymbol{\omega}/\|\boldsymbol{\omega}\|)$, which theoretically achieves a similar effect.

A more advanced approach combines this with the steepest descent idea to adjust the update rule, as discussed in articles \href{https://kexue.fm/archives/11196}{\emph{Steepest Descent on Manifolds: 1. SGD + Hypersphere}} and \href{https://kexue.fm/archives/11241}{\emph{Steepest Descent on Manifolds: 4. Muon + Spectral Sphere}}, and in the paper \href{https://papers.cool/arxiv/2601.08393}{\emph{Controlled LLM Training on Spectral Sphere}}. This approach is more elegant in principle but more complex in practice, often requiring solving a nonlinear equation to obtain the exact update.

However, should we really insist on strictly controlling a parameter's norm to a fixed value? Intuitively, the parameter norm should be determined by the training process itself; at best, we set a prior range for it. Although some work shows that setting an appropriate fixed norm does not harm performance, it still disrupts the original training dynamics and may require extra effort to adapt.

Therefore, the view proposed in this article is that we only need to ensure $\|\boldsymbol{\omega}\| = \mathcal{O}(\tau)$. Specifically, we aim to guarantee $\|\boldsymbol{\omega}\| \leq \tau$ at every step, while leaving the exact value—including whether it reaches $\Theta(\tau)$—to be determined by the training algorithm, without further intervention.

\section{Post Clip}
\label{Post Clip}

The next question is: how to achieve $\|\boldsymbol{\omega}\| \leq \tau$? More specifically, suppose the original update rule for $\boldsymbol{\omega}$ is
\begin{equation}
\boldsymbol{\omega}_t = \boldsymbol{\omega}_{t-1} - \eta \boldsymbol{\phi}_t
\label{eq:base-update}
\end{equation}
How should we modify it to ensure $\|\boldsymbol{\omega}_t\| \leq \tau$? There are many possible methods; for instance, normalization as mentioned earlier is one option. Among them, we aim to select the one that causes the \emph{least disruption} to the optimization process. That is, given $\boldsymbol{\omega}$ and a norm $\|\cdot\|$, we aim to minimize the modification to make its norm at most $\tau$. Formally, this is defined as:
\begin{equation}
\left\lfloor\boldsymbol{\omega}\right\rfloor_{\|\cdot\|\leq\tau} = \operatorname{argmin}_{\|\tilde{\boldsymbol{\omega}}\|\leq\tau} \|\boldsymbol{\omega} - \tilde{\boldsymbol{\omega}}\|_{RMS}
\label{eq:nclip}
\end{equation}
Familiar readers of convex optimization will recognize this as the projection of $\boldsymbol{\omega}$ into a ball of radius $\tau$ under the specified norm. The key idea is to achieve the goal of bounded norm while minimizing the perturbation to the original parameter $\boldsymbol{\omega}$, thus inducing a specific projection or clipping operation.

The exact computation of $\left\lfloor\boldsymbol{\omega}\right\rfloor_{\|\cdot\|\leq\tau}$ depends on the specific norm and will be detailed later. With this operator, one possible scheme is to apply it after each update:
\begin{equation}
\boldsymbol{\omega}_t = \left\lfloor\boldsymbol{\omega}_{t-1} - \eta \boldsymbol{\phi}_t\right\rfloor_{\|\cdot\|\leq\tau}
\end{equation}
We call this scheme ``Post Clip,'' which is simple and intuitive. However, it may feel ``non-smooth'': if initialization starts below $\tau$, the parameter norm increases slowly until reaching $\tau$, at which point clipping ``suddenly'' activates. While continuous, this is not differentiable at the boundary—similar to the $\max(x,0)$ function.

\section{Pre Decay}
\label{Pre Decay}

If this non-smoothness is undesirable, one can imitate weight decay by distributing the penalty across updates. Still starting from update rule~\eqref{eq:base-update}, suppose $\|\boldsymbol{\phi}_t\| \leq \tau$, then by the triangle inequality:
\begin{equation}
\|\boldsymbol{\omega}_t\| = \|\boldsymbol{\omega}_{t-1} - \eta \boldsymbol{\phi}_t\| \leq \|\boldsymbol{\omega}_{t-1}\| + \eta \tau,
\end{equation}
meaning the norm could grow by up to $\eta\tau$ per step, accumulating out of control over time.

To prevent this, we can pre-process $\boldsymbol{\omega}_{t-1}$ before subtracting $\eta\boldsymbol{\phi}_t$, reducing its norm just enough to counteract the potential increase. Following the weight decay analogy:
\begin{equation}
\boldsymbol{\omega}_t = \left\lfloor\boldsymbol{\omega}_{t-1}\right\rfloor_{\|\cdot\|\leq (1-\eta)\|\boldsymbol{\omega}_{t-1}\|} - \eta \boldsymbol{\phi}_t
\label{eq:pre-decay}
\end{equation}
That is, first reduce $\boldsymbol{\omega}_{t-1}$'s norm to $(1-\eta)$ times its original value, then update. This gives:
\[
\|\boldsymbol{\omega}_t\| \leq (1-\eta)\|\boldsymbol{\omega}_{t-1}\| + \eta \tau \leq \max(\|\boldsymbol{\omega}_{t-1}\|,\tau),
\]
and by induction, $\|\boldsymbol{\omega}_t\| \leq \max(\|\boldsymbol{\omega}_0\|,\tau)$. Thus, if initialization satisfies $\|\boldsymbol{\omega}_0\| \leq \tau$, then $\|\boldsymbol{\omega}_t\| \leq \tau$ holds throughout training. This result is independent of the choice of norm—it only relies on the triangle inequality. The minimal-modification operation to reduce norm is precisely the clipping operator in~\eqref{eq:nclip}, making its use natural here.

We call this ``Pre Decay.'' Unlike Post Clip, which uses a static threshold $\tau$, Pre Decay uses a dynamic threshold $(1-\eta)\|\boldsymbol{\omega}_{t-1}\|$ and always triggers a reduction, making the process smoother. Hence, we call it ``decay'' instead of ``clip,'' generalizing conventional weight decay.

\section{Simple Example}
\label{Simple Example}

So far, we have established a general framework for constraining parameter norms, with Post Clip and Pre Decay as two schemes, centered on the clipping operator $\left\lfloor\boldsymbol{\omega}\right\rfloor_{\|\cdot\|\leq\tau}$ defined in~\eqref{eq:nclip}. While formally defined, practical implementation depends on the norm.

Consider a simple example using $\|\cdot\|_{RMS}$. For vectors, this equals the L2 norm; for matrices, it equals the Frobenius norm. We have:
\begin{equation}
\left\lfloor\boldsymbol{\omega}\right\rfloor_{\|\cdot\|_{RMS}\leq\tau} = \operatorname{argmin}_{\|\tilde{\boldsymbol{\omega}}\|_{RMS}\leq\tau} \|\boldsymbol{\omega} - \tilde{\boldsymbol{\omega}}\|_{RMS} = \min\left(1,\,\frac{\tau}{\|\boldsymbol{\omega}\|_{RMS}}\right)\boldsymbol{\omega}
\end{equation}
The proof is left to the reader (if stuck, consult Kimi). Substituting $\tau = (1 - \eta) \|\boldsymbol{\omega}\|_{RMS}$ yields:
\begin{equation}
\left\lfloor\boldsymbol{\omega}\right\rfloor_{\|\cdot\|_{RMS}\leq (1 - \eta) \|\boldsymbol{\omega}\|_{RMS}} = \min\left(1,\,\frac{(1 - \eta) \|\boldsymbol{\omega}\|_{RMS}}{\|\boldsymbol{\omega}\|_{RMS}}\right)\boldsymbol{\omega} = (1-\eta)\boldsymbol{\omega}
\end{equation}
Plugging into~\eqref{eq:pre-decay} gives:
\begin{equation}
\boldsymbol{\omega}_t = (1-\eta)\boldsymbol{\omega}_{t-1} - \eta \boldsymbol{\phi}_t
\end{equation}
This is precisely conventional weight decay. Thus, Pre Decay under RMS norm reduces to standard weight decay, the minimal modification scheme to preserve RMS (equivalently, L2 or Frobenius) norm bounds.

\section{Singular Clipping}
\label{Singular Clipping}

We now turn to the main event—matrix parameters and Muon. Recall the Muon update:
\begin{equation}
\boldsymbol{W}_t = \boldsymbol{W}_{t-1} - \eta\lambda\boldsymbol{\Phi}_t,\quad \boldsymbol{\Phi}_t=\frac{1}{\lambda}\sqrt{\frac{d_{out}}{d_{in}}}\msign(\boldsymbol{G}_t)
\end{equation}
Let $\tau = \frac{1}{\lambda}\sqrt{\frac{d_{out}}{d_{in}}}$, so $\|\boldsymbol{\Phi}_t\|_2 = \tau$. To ensure $\|\boldsymbol{W}_t\|_2 \leq \tau$, two schemes are:
\begin{align}
\text{Post Clip:}\quad\boldsymbol{W}_t &= \left\lfloor\boldsymbol{W}_{t-1} - \eta\lambda\boldsymbol{\Phi}_t\right\rfloor_{\|\cdot\|_2\leq\tau} \\[5pt]
\text{Pre Decay:}\quad\boldsymbol{W}_t &= \left\lfloor\boldsymbol{W}_{t-1}\right\rfloor_{\|\cdot\|_2\leq(1-\eta\lambda)\|\boldsymbol{W}_{t-1}\|_2}  - \eta\lambda\boldsymbol{\Phi}_t
\end{align}
Now compute $\left\lfloor\boldsymbol{W}\right\rfloor_{\|\cdot\|_2\leq\tau}$, which is:
\begin{equation}
\left\lfloor\boldsymbol{W}\right\rfloor_{\|\cdot\|_2\leq\tau} = \operatorname{argmin}_{\|\tilde{\boldsymbol{W}}\|_2\leq\tau} \|\boldsymbol{W} - \tilde{\boldsymbol{W}}\|_F
\label{eq:mclip-loss}
\end{equation}
This problem's optimal solution is known as ``Singular Value Clipping (SVC)'' or $\mclip$:
\begin{equation}
\left\lfloor\boldsymbol{W}\right\rfloor_{\|\cdot\|_2\leq\tau} = \mclip(\boldsymbol{W};\tau) = \boldsymbol{U}\min(\boldsymbol{\Sigma},\tau)\boldsymbol{V}^{\top}
\label{eq:2-to-mclip}
\end{equation}
where $\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top}$ is the SVD of $\boldsymbol{W}$ and $\min(\boldsymbol{\Sigma},\tau)$ truncates singular values to at most $\tau$. With this notation:
\begin{align}
\text{Post Clip:}\quad\boldsymbol{W}_t &= \mclip(\boldsymbol{W}_{t-1} - \eta\lambda\boldsymbol{\Phi}_t;\tau) \\[5pt]
\text{Pre Decay:}\quad\boldsymbol{W}_t &= \mclip(\boldsymbol{W}_{t-1};(1-\eta\lambda)\|\boldsymbol{W}_{t-1}\|_2)  - \eta\lambda\boldsymbol{\Phi}_t
\end{align}

\section{Derivation}
\label{Derivation}

We now prove~\eqref{eq:2-to-mclip}. Let $\boldsymbol{W} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top}$ be the SVD. Then:
\begin{align}
\|\boldsymbol{W} - \tilde{\boldsymbol{W}}\|_F &= \|\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top} - \tilde{\boldsymbol{W}}\|_F \\
&= \|\boldsymbol{U}^{\top}(\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top} - \tilde{\boldsymbol{W}})\boldsymbol{V}\|_F \\
&= \|\boldsymbol{\Sigma} - \boldsymbol{U}^{\top}\tilde{\boldsymbol{W}}\boldsymbol{V}\|_F
\end{align}
since orthogonal transformations preserve Frobenius norm. Let $\tilde{\boldsymbol{\Sigma}} = \boldsymbol{U}^{\top}\tilde{\boldsymbol{W}}\boldsymbol{V}$. The objective becomes:
\begin{equation}
\operatorname{argmin}_{\|\tilde{\boldsymbol{\Sigma}}\|_2\leq\tau} \|\boldsymbol{\Sigma} - \tilde{\boldsymbol{\Sigma}}\|_F
\end{equation}
Since $\|\tilde{\boldsymbol{\Sigma}}\|_2 \leq \tau$ and the minimum of $\|\boldsymbol{\Sigma} - \tilde{\boldsymbol{\Sigma}}\|_F$ occurs when off-diagonal elements of $\tilde{\boldsymbol{\Sigma}}$ are zero and diagonal elements are $\tilde{\sigma}_i = \min(\sigma_i,\tau)$, we obtain:
\begin{equation}
\tilde{\boldsymbol{\Sigma}}^* = \min(\boldsymbol{\Sigma},\tau)
\end{equation}
Thus, $\tilde{\boldsymbol{W}}^* = \boldsymbol{U}\min(\boldsymbol{\Sigma},\tau)\boldsymbol{V}^{\top}$, proving~\eqref{eq:2-to-mclip}.

\section{Main Component Clipping}
\label{Main Component Clipping}

How to efficiently compute $\mclip$? Performing SVD every step is expensive. As discussed in~\href{https://kexue.fm/archives/11006}{\emph{Computing mclip via msign (Part I)}} and~\href{https://kexue.fm/archives/11059}{\emph{Computing mclip via msign (Part II)}}, one can use $\msign$ but it requires multiple passes. An alternative uses iterative top-singular-value estimation ($\SVDone$). Since $\mclip$ clips all singular values above $\tau$ to $\tau$, repeatedly clipping the top singular value achieves the same effect. Assuming training smoothness, one clipping per step suffices. Thus:
\begin{align}
\text{Post Clip:}\quad\boldsymbol{W}_t &= \tilde{\boldsymbol{W}}_t - \max(\sigma_1 - \tau, 0) \boldsymbol{u}_1 \boldsymbol{v}_1^{\top},\quad\sigma_1, \boldsymbol{u}_1, \boldsymbol{v}_1 = \SVDone(\tilde{\boldsymbol{W}}_t),\quad\tilde{\boldsymbol{W}}_t = \boldsymbol{W}_{t-1} - \eta \boldsymbol{\Phi}_t \\[5pt]
\text{Pre Decay:}\quad\boldsymbol{W}_t &= \boldsymbol{W}_{t-1} - \lambda\eta\sigma_1 \boldsymbol{u}_1 \boldsymbol{v}_1^{\top} - \eta \boldsymbol{\Phi}_t,\quad\sigma_1, \boldsymbol{u}_1, \boldsymbol{v}_1 = \SVDone(\boldsymbol{W}_{t-1})
\end{align}
The Pre Decay version matches the spectral weight decay introduced in~\href{https://kexue.fm/archives/10648}{\emph{From Spectral Gradient to New Weight Decay}}.

\section{Other Details}
\label{Other Details}

For greater accuracy, approximate $k$ largest singular values and clip up to $k$ per step. This requires QR during power iteration (see~\href{https://kexue.fm/archives/11654}{\emph{Streamed Power Iteration for Muon: Part I}}).

Other norms appear in~\href{https://kexue.fm/archives/11647}{\emph{Beyond MuP: 3}}, e.g., row/column RMS for Embedding/LM Head and $\ell_\infty$ norm for gamma in RMS Norm. Their clipping operators are straightforward:
- Embedding: clip each row's RMS to $\leq\tau$
- LM Head: same for columns
- gamma: clip element-wise to $[-\tau,\tau]$, i.e., $\clip(\boldsymbol{\gamma};\tau,-\tau)$

These are intuitive and left as exercises.

\section{Necessary Justification}
\label{Necessary Justification}

Some may ask: why not use ordinary weight decay as in~\href{https://papers.cool/arxiv/2502.07529}{\emph{Training Deep Learning Models with Norm-Constrained LMOs}}:
\begin{equation}
\boldsymbol{W}_t = (1-\eta\lambda)\boldsymbol{W}_{t-1}  - \eta\sqrt{\frac{d_{out}}{d_{in}}}\msign(\boldsymbol{G}_t)
\end{equation}
to limit spectral norm?

The issue is over-intervention. The clipping operator~\eqref{eq:nclip} minimizes perturbation. Multiplying by $1-\eta\lambda$ reduces spectral norm but is not minimal; hence, it over-intervenes. Consequences:
- Small $\lambda$: insufficient control ($\tau$ too large)
- Large $\lambda$: too much damping

For example, to bound spectral norm by 5 with $d_{in}=d_{out}$, $\lambda=0.2$ gives excessive 0.2 weight decay (typical is 0.01).

We stress ``guarantee'': small models may stay below 5 with decay 0.01, but large models can reach theoretical limits (here 100). Maintaining a tight theoretical bound is essential for safety—this is the ``stable'' in ``stable acceleration''. The clipping operator~\eqref{eq:nclip} provides the lightest intervention that guarantees bounded norms, thus minimizing performance loss.

\section{Summary}
\label{Summary}

This article, based on minimal-perturbation principles, proposes a general framework for maintaining parameter stability during training, including Post Clip and Pre Decay. Under spectral norm, they induce Singular Value Clipping and Spectral Weight Decay. These operations aim to bound key parameter norms while minimizing interference with training dynamics.
\end{document}
