# A Guide to the Muon Optimizer: Quick Start and Key Details Source: Machine translated. Refer to the original post to verify accuracy. --- author: Jianlin Su date: November 19, 2025 lang: en markdown_url: 11416-Muon-Optimizer-Guide-Quick-Start-and-Key-Details.md next_url: 11404-Asymptotic-Estimation-of-Weight-RMS-for-AdamW-Part-2.html post_id: 11416 prev_url: 11428-Generative-Diffusion-Models-31-Predicting-Data-Instead-of-Noise.html site_title: English (unofficial) translations of posts at kexue.fm source_url: "https://kexue.fm/archives/11416" title: "A Guide to the Muon Optimizer: Quick Start and Key Details" --- Recently, I believe many readers have already come across news about the Muon optimizer in their feeds. In fact, Muon was proposed around October of last year by [Keller Jordan](https://x.com/kellerjordan0/status/1842300916864844014) on Twitter, so only a little more than a year has passed since then. Yet, within just this one year, Muon has already withstood the test of training models with tens of billions, hundreds of billions, and even trillions of parameters, which is enough to show that it is a highly competitive optimizer. Today, Muon is already built into training frameworks such as [Torch](https://docs.pytorch.org/docs/stable/generated/torch.optim.Muon.html) and [Keras](https://keras.io/api/optimizers/muon/), and even large frameworks like [Megatron](https://github.com/NVIDIA/Megatron-LM/blob/dev/megatron/core/optimizer/muon.py) have gradually begun to support it, which means it has already gained widespread recognition in the industry. However, for readers who are only familiar with Adam, how to switch to Muon quickly and effectively may still be a confusing matter. Therefore, this article attempts to provide a quick-start tutorial. # A Brief Introduction The formal proposer of Muon is [Keller Jordan](https://x.com/kellerjordan0/status/1842300916864844014), who currently works at OpenAI. As mentioned at the beginning, Muon was first published on Twitter, and up to now the author has merely added a blog post, [*Muon: An Optimizer for Hidden Layers in Neural Networks*](https://kellerjordan.github.io/posts/muon/), rather than a formal paper; in the author’s view, “whether it is written up as a paper has absolutely nothing to do with whether the optimizer is effective”\[[original text](https://x.com/kellerjordan0/status/1890178773586489716)\]. Muon is an optimizer customized specifically for matrix parameters. There are also related works with similar characteristics, such as [Shampoo](https://papers.cool/arxiv/1802.09568), and the somewhat earlier [Stochastic Spectral Descent](./10592-An-Appreciation-of-the-Muon-Optimizer-The-Essential-Leap-from-Vectors-to-Matrices.html), among others. Many works can be related to Muon to a greater or lesser extent, but none of them fully covers Muon, so in the author’s view Muon counts as a brand-new piece of work. Within China, the earliest article popularizing Muon was probably the author’s blog post [*An Appreciation of the Muon Optimizer: The Essential Leap from Vectors to Matrices*](./10592-An-Appreciation-of-the-Muon-Optimizer-The-Essential-Leap-from-Vectors-to-Matrices.html), and the first validation of Muon on a relatively large-scale model was [Moonlight](https://papers.cool/arxiv/2502.16982), which we released in February; the Moonlight version of Muon proposed therein was used in the subsequent trillion-parameter [K2](https://papers.cool/arxiv/2507.20534). After K2, [GLM-4.5](https://papers.cool/arxiv/2508.06471) likewise adopted this Muon variant. As Jeremy Bernstein, one of Muon’s authors, says in his blog post [*Deriving Muon*](https://jeremybernste.in/writing/deriving-muon), for the author, the distinctive feature of Muon is that it can be derived from more fundamental optimization principles while remaining effective in practice; by contrast, although Adam is also very effective, it is more like a heuristic scheme. # Four Versions This article does not intend to introduce the mathematical details of Muon, nor its implementation; instead, it mainly covers some technical details and caveats involved in switching from Adam to Muon. As just mentioned, Muon is dedicated to the optimization of matrix parameters and uses a non-element-wise update rule, which may make it somewhat confusing for new users to get started. Moreover, to the best of the author’s knowledge, Muon currently has at least four slightly different versions, and this multi-version situation further aggravates the confusion. If users are unaware of the details, they may get poor results due to incorrectly tuned hyperparameters (especially the learning rate); below we will focus on getting all of this straight. First, for a matrix $`\boldsymbol{W}\in\mathbb{R}^{d_{\mathrm{in}}\times d_{\mathrm{out}}}`$, with $`\boldsymbol{G}`$ denoting its gradient, the four Muon variants are: ``` math \begin{aligned} &\quad\boldsymbol{M}_t \;=\; \beta \boldsymbol{M}_{t-1} + \boldsymbol{G}_t \\[10pt] &\quad\boldsymbol{W}_t \;=\; \left\{ \begin{aligned} &\boldsymbol{W}_{t-1} - \eta_t \left(\mathop{\mathrm{msign}}(\boldsymbol{M}_t) + \lambda \boldsymbol{W}_{t-1}\right) & & \textcolor{skyblue}{(\text{naive version})} \\[5pt] &\boldsymbol{W}_{t-1} - \eta_t \left(\sqrt{\max(1, d_{\mathrm{out}}/d_{\mathrm{in}})}\,\mathop{\mathrm{msign}}(\boldsymbol{M}_t) + \lambda \boldsymbol{W}_{t-1}\right) & & \textcolor{skyblue}{(\text{KellerJordan version})} \\[5pt] &\boldsymbol{W}_{t-1} - \eta_t \left(\sqrt{d_{\mathrm{out}}/d_{\mathrm{in}}}\,\mathop{\mathrm{msign}}(\boldsymbol{M}_t) + \lambda \boldsymbol{W}_{t-1}\right) & & \textcolor{skyblue}{(\text{MuP version})} \\[5pt] &\boldsymbol{W}_{t-1} - \eta_t \left(0.2\times\sqrt{\max(d_{\mathrm{out}},d_{\mathrm{in}})}\,\mathop{\mathrm{msign}}(\boldsymbol{M}_t) + \lambda \boldsymbol{W}_{t-1}\right) & & \textcolor{skyblue}{(\text{Moonlight version})} \end{aligned} \right. \end{aligned} ``` To enable Nesterov momentum, replace $`\mathop{\mathrm{msign}}(\boldsymbol{M}_t)`$ with $`\mathop{\mathrm{msign}}(\beta\boldsymbol{M}_t + \boldsymbol{G}_t)`$; here $`\mathop{\mathrm{msign}}`$ is usually named `zeropower_via_newtonschulz` in implementations, and ordinary users can ignore the concrete implementation details. The only difference among the four versions is the scaling factor in front of $`\mathop{\mathrm{msign}}`$: the “KellerJordan version” and the “MuP version” are broadly alike, while the “Moonlight version” is a bit special. Keras implements only the “KellerJordan version,” while Torch implements both the “KellerJordan version” and the “Moonlight version”; the naive version seems to be relatively rare at present, and the one the author commonly uses is a self-written “MuP version.” # Two Dimensions Here we must pay attention to an important detail: the “KellerJordan version” and the “MuP version” are sensitive to the order of $`d_{\mathrm{in}}`$ and $`d_{\mathrm{out}}`$, so the first thing to do is to figure out what $`d_{\mathrm{in}}`$ and $`d_{\mathrm{out}}`$ mean; it is not necessarily the case that the first dimension of the matrix is $`d_{\mathrm{in}}`$ and the second dimension is $`d_{\mathrm{out}}`$. $`d_{\mathrm{in}}`$ and $`d_{\mathrm{out}}`$ denote the input and output dimensions of the linear layer respectively, so which one is $`d_{\mathrm{in}}`$ and which one is $`d_{\mathrm{out}}`$ depends on the specific implementation of the linear layer. For instance, Keras’s Dense layer computes $`\boldsymbol{x}\boldsymbol{W}`$, so the first dimension of the matrix $`\boldsymbol{W}`$ is $`d_{\mathrm{in}}`$ and the second is $`d_{\mathrm{out}}`$; however, Torch’s Linear layer computes $`\boldsymbol{x}\boldsymbol{W}^{\top}`$, so the second dimension of the matrix $`\boldsymbol{W}`$ is $`d_{\mathrm{in}}`$, and the first dimension is $`d_{\mathrm{out}}`$. Therefore, to implement the “KellerJordan version” of Muon, for Torch’s Linear layer the scaling factor should be `max(1, W.shape[0]/W.shape[1])**0.5`, whereas for Keras it should be `max(1, W.shape[1]/W.shape[0])**0.5`. Consequently, the Muon implementation in the current version of Keras (3.12) is actually incorrect, because it copies Torch’s scaling-factor implementation verbatim\[[source code](https://github.com/keras-team/keras/blob/v3.12.0/keras/src/optimizers/muon.py)\]. If the model is written by yourself, you need to judge carefully according to your own code; for example, one cannot rule out the possibility of mixing Torch’s built-in Linear layers with hand-written `x @ W`, in which case there is no single answer as to whether it should be `W.shape[0]/W.shape[1]` or `W.shape[1]/W.shape[0]`. Of course, if you find sorting all of this out too much trouble, you can consider using the “Moonlight version,” whose scaling factor is symmetric with respect to $`d_{\mathrm{in}}`$ and $`d_{\mathrm{out}}`$. # Hyperparameter Settings With $`d_{\mathrm{in}}`$ and $`d_{\mathrm{out}}`$ sorted out, what remains is how to set the learning rate $`\eta_t`$ and the weight decay coefficient $`\lambda`$. The assumption here is that the user already has experience tuning Adam and has achieved decent results under Adam, and now wants to quickly migrate to Muon to give it a try. Let us first look at the “Moonlight version.” Its scaling factor is obtained by aligning with Adam’s Update RMS; if you want to understand the details, you can refer to [*The Muon Sequel: Why We Chose to Try Muon?*](./10739-Muon-Sequel-Why-We-Chose-to-Try-Muon.html), and as for the “magic number” $`0.2`$, you can refer to [*Why Is Adam’s Update RMS 0.2?*](./11267-Why-is-Adam-s-Update-RMS-0.2.html). Simply put, “Moonlight version” Muon is aligned with Adam’s update magnitude, so the simplest way to migrate over from Adam is: **change nothing at all**—just keep using Adam’s $`\eta_t`$ and $`\lambda`$. Now consider the remaining three versions. As we know, mainstream models usually have a `hidden_size` (denoted $`d`$), and most of the model’s matrix shapes do not deviate noticeably from $`d\times d`$, so we treat things approximately with $`d_{\mathrm{in}}=d_{\mathrm{out}}=d`$. Under this approximation, the three versions are all identical, differing from the “Moonlight version” only by a missing factor of $`0.2\sqrt{d}`$. Since the “Moonlight version” aligns with Adam’s update magnitude and hence allows keeping the hyperparameters unchanged, the learning rates of these three versions should be scaled up by a factor of $`0.2\sqrt{d}`$ in order to align with Adam’s update magnitude, and correspondingly, $`\lambda`$ should be divided by $`0.2\sqrt{d}`$. Substituting $`d=1024, 2048, 4096`$ gives $`6.4`$, $`9`$, and $`12.8`$ respectively. If you cannot remember $`0.2\sqrt{d}`$, then you can simply remember that if we use one of the other three versions of Muon, we should **directly scale Adam’s learning rate up by a factor of 10** to serve as Muon’s learning rate. If Adam’s learning rate is plugged directly into Muon, one will conclude—because of underfitting—that Muon is far inferior to Adam; as far as the author knows, some of Muon’s bad reviews originate exactly from this. Seen this way, is the “Moonlight version” still the better choice after all? The “Moonlight version” does deliver good practical results, but to declare it more convenient on that basis is to evaluate things from Adam’s standpoint. The advantage of the “MuP version” or the “KellerJordan version” is that the learning rate is transferable: once the learning rate has been tuned on a small model, using it directly on a large model often still gives decent results. For this part, refer to Jeremy Bernstein’s blog post [*Deriving Muon*](https://jeremybernste.in/writing/deriving-muon) or the author’s blog post [*Higher-Order MuP: A Simpler yet More Ingenious Spectral Condition Scaling*](./10795-Higher-Order-MuP-A-Simpler-Yet-Smarter-Spectral-Condition-Scaling.html). # Other Parameters If Muon only takes care of matrix parameters, then what about the remaining parameters? For example, the bias term of a linear layer and the gamma term of RMSNorm are 1-dimensional parameters; convolutional layers, in turn, may involve parameters in 3- or 4-dimensional arrays. A correction is in order here first: Muon is not merely about matrix parameters; Muon takes care only of “**the matrix parameters of linear layers with dense inputs**.” If readers find this rather hard to parse, just remember that the matrix parameters of the Embedding layer and of the final classification layer (including GPT’s LM Head) must not use Muon, otherwise the results will be noticeably worse. For these matrix parameters that cannot use Muon, as well as 1-dimensional, 3-dimensional, and higher-dimensional parameters, if readers do not want to spend too much effort, simply use Adam—basically every Muon implementation is mixed with Adam, and users can choose to use Adam for certain layers. If readers are willing to tinker, then parameters with 3 or 4 dimensions, such as those of convolutional layers, can also use Muon. Take Conv2D as an example: the convolution kernel usually has shape $`(w, h, d_{\mathrm{in}}, d_{\mathrm{out}})`$, and its equivalent implementation actually flattens the $`(w, h, d_{\mathrm{in}})`$ patch input into a vector of size $`w \times h \times d_{\mathrm{in}}`$, then reshapes the convolution kernel into $`(w\times h \times d_{\mathrm{in}}, d_{\mathrm{out}})`$ as well, and finally performs a matrix multiplication. Hence, to use Muon here, one first reshapes the momentum into $`(w\times h \times d_{\mathrm{in}}, d_{\mathrm{out}})`$, computes $`\mathop{\mathrm{msign}}`$, and then reshapes it back for the update. Similarly, for the gamma parameter of RMSNorm, it can be viewed as multiplication by a diagonal matrix, so by treating its momentum as a diagonal matrix one can likewise compute $`\mathop{\mathrm{msign}}`$, and the result is equivalent to SignSGDM; the Embedding layer can be viewed as computing $`\mathop{\mathrm{msign}}`$ for multiple $`(1,d)`$ matrices, and the result is Normalized SGDM (refer to [*An Appreciation of the Muon Optimizer: The Essential Leap from Vectors to Matrices*](./10592-An-Appreciation-of-the-Muon-Optimizer-The-Essential-Leap-from-Vectors-to-Matrices.html)). If you still want to experiment further—for example, for Multi-Head Attention, could the projection matrix of each head be taken out separately and given its own $`\mathop{\mathrm{msign}}`$ … As long as life goes on, the tinkering never stops~ # Expected Results Finally, if the user has set everything up correctly according to the instructions above and got it running, then it is time to start praying for the goddess of luck to descend. What kind of result should we expect? If anomalies such as gradient explosions do not occur, then in most cases Muon will be slightly better than Adam; of course, one cannot rule out that in some situations Muon will be slightly worse, but either way, the gap between them will not be very large. If one side turns out to be far better than the other, then it is probably worth reflecting on which side’s setup has gone wrong. However, none of this is absolute; for example, under certain extreme settings, it can indeed happen that Muon is much better than Adam while Adam cannot be rescued no matter how it is tuned. In a word, good luck to you. If you observe any interesting phenomena, you are welcome to discuss and analyze them together. ***When reposting this article, please include its address:** [https://kexue.fm/archives/11416](./11416-Muon-Optimizer-Guide-Quick-Start-and-Key-Details.html)* ***For more detailed reposting policies, please refer to:*** [*Scientific Spaces FAQ*](./06508-Scientific-Space-Browsing-Guide-FAQ.html)