Optimizer taxonomy + benchmark

OmniOpt

Taxonomy, Geometry, and Benchmarking of Modern Optimizers

A unified meta-pipeline, LMO-grounded geometry, and cross-domain benchmark for choosing modern optimizers under compute, memory, stability, robustness, and generalization constraints.

Siyuan Li1,3,†, Jiabao Pan1,2,†, Yumou Liu1,4,†, Zhuoli Ouyang7,†, Xin Jin3,†, Xinglong Xu5, Jingxuan Wei5, Shengye Pang2,*, Jintao Chen6, Xuanhe Zhou4, Conghui He1, Cheng Tan1,*

1Shanghai Artificial Intelligence Laboratory · 2Shanghai University · 3Westlake University · 4Shanghai Jiao Tong University · 5UCAS · 6Zhejiang University · 7Southern University of Science and Technology

†Equal contribution. *Corresponding authors.

Code Dataset Benchmark Tables Cite
Timeline showing optimizer design evolving from a single AdamW-style default into a diversified mechanism space across T1-T5 families
Figure 2. Optimizer design has moved from a single AdamW-style default into a diversified mechanism space. This overview is not a leaderboard; it frames why the paper needs a shared taxonomy, geometry, and benchmark protocol.
100+optimizer methods organized
24benchmarked optimizers
60M-1Blanguage model scales
4 arch.standard and linear-attention models
Framework

One optimizer step as a five-stage transformation.

The paper treats optimizers as structured transformations across parameter routing, gradient transformation, state evolution, update reconstruction, and finalization. Most methods do meaningful work in only one or two stages, making comparison and composition easier.

Overview of the meta-pipeline, LMO framework, dual-dimension taxonomy, and benchmark connecting the paper's four components
Figure 1. Overview of the proposed survey and benchmark framework for a wide range of optimizers. The paper first introduces a universal meta-pipeline, then develops an LMO-driven four-axis decomposition, builds a process-aligned methodological taxonomy and an effect-objective taxonomy, and finally connects both taxonomies to a large-scale benchmark study.
Universal meta-pipeline for a modern optimizer update
Figure 3. Universal meta-pipeline for one optimizer step. The figure is shown full-width because the stage labels and tensor routes are the central visual evidence in this section.
S0

Signal acquisition

Receives first-order gradients, variance-reduced signals, or curvature-augmented estimates from the training system.

S1

Parameter routing

Partitions tensors by shape and module type so matrices, vectors, heads, or layers can follow different update routes.

S2

Gradient transform

Applies the mechanism that changes direction space: identity maps, sign maps, spectral orthogonalization, Kronecker transforms, or low-rank projection.

S3

State evolution

Maintains moment, curvature, factorized, quantized, or variance-reduced states before a direction is formed.

S4

Reconstruction

Returns transformed or compressed directions to the full parameter space through inverse rotations, projections, or approximations.

S5

Finalization

Writes the update with learning rate, weight decay, clipping, trust ratios, masks, or sharpness-aware corrections.

Bridge to the LMO / four-axis geometry

The meta-pipeline locates where an optimizer intervenes. The geometric view explains what direction that intervention creates: state estimation happens before geometry, and the LMO or preconditioner consumes the estimated state to form the update.

Axis I

Update domain

Where the update lives: full parameter space, matrix space, rotated coordinates, or a low-rank subspace.

Axis II

State estimator

How momentum, second moments, Gram/Hessian proxies, variance reduction, and projection state are produced.

Axis III

Geometry operator

How the state becomes a direction through an LMO constraint set or a Hessian-style preconditioner.

Axis IV

Finalization wrapper

How learning rate, decay, projection-back, routing fallbacks, refresh schedules, and clipping commit the direction.

Representative optimizer families viewed through the universal meta-pipeline. Only active stages and defining mechanisms are shown; inactive stages follow identity maps or standard defaults.
Method (family) Active stages Core mechanism
AdamW (T1.1) S3, S5 Moment EMAs (S3) with decoupled weight decay (S5)
Muon (T2.1) S1, S2 Matrix routing (S1) with Newton-Schulz spectral orthogonalization (S2)
GaLore (T2.3) S1-S4 Low-rank projection (S1/S2), subspace Adam state (S3), and inverse projection (S4)
Lion (T3) S2, S3 Momentum interpolation (S3) followed by sign discretization (S2)
SAM (T5.1) S0, S5 Perturbation-induced gradient (S0) with neighborhood-regularized writeback (S5)
Dual taxonomy

Method families meet effect objectives.

The page preserves both taxonomy axes: mechanism families T1-T5 and effect objectives O1-O6. This is the map used to interpret benchmark tradeoffs rather than a leaderboard-only view.

T1

Element-wise adaptive moment

Adam-style scalar control and moment estimation.

T2

Matrix-structured methods

Spectral, Kronecker, and subspace update directions.

T3

Discretized directions

Sign-like and quantized update geometry.

T4/T5

Compression and geometry

State reduction, curvature, perturbation, and trust-region controls.

Mechanism overview of T1 element-wise adaptive-moment and scalar control methods
Figure 4. Mechanism overview of T1 element-wise adaptive-moment and scalar control methods. The family is organized by the dominant intervention in the update: direct scalar or Adam-style variants, additional temporal or estimator-correction channels, and outer iterate or global-scale control.
Mechanism schematic for T2 matrix-level structural methods
Figure 5. Mechanism schematic for T2 matrix-level structural methods. The schematic summarizes the three matrix routes used in this survey: spectral direction selection, Kronecker or structured-Fisher preconditioning, and low-rank subspace projection.
Mechanism schematic for T3 discretization and directional quantization methods
Figure 6. Mechanism schematic for T3 discretization and directional quantization. This schematic groups discretization and directional quantization mechanisms by their sign-direction generation methods.
Mechanism schematic for T4 state compression and structural aggregation methods
Figure 7. Mechanism schematic for T4 state compression and structural aggregation. The schematic organizes the family by the form of memory reduction: factored second-moment storage, low-bit state representation, shared adaptive statistics, and streaming gradient consumption.
Mechanism schematic for T5 curvature-aware and geometric regularization methods
Figure 8. Mechanism schematic for T5 curvature-aware and geometric regularization methods. The schematic organizes the family by where the geometric intervention enters the update: perturbed-gradient evaluation, diagonal-curvature estimation, post-update filtering, or layer-wise trust-region scaling.
Dimension-B effect objectives and measurement sources.
ID Name Definition Data source Extra cost Typical outputs
O1 Convergence Efficiency Loss reduction and time-to-target under a fixed training budget Train/validation loss logs None Final loss; steps-to-threshold; token efficiency
O2 Step cost Extra per-step computation and synchronization relative to a baseline Timers, FLOP analysis Recorded during training Step time; FLOPs; extra backward count
O3 Memory Memory from optimizer states and associated buffers Memory profiler, byte model Recorded during training Peak memory; state and buffer bytes
O4 Stability Robustness to spikes, divergence, and gradient fluctuations Loss and gradient-norm curves Offline post-processing Spike rate; gradient CV; divergence rate
O5 Hparam robustness Sensitivity to learning rate, decay, batch size, and other knobs Multiple training runs Search or transfer experiments Usable LR interval; performance variance; tuning burden
O6 Generalization Quality beyond the training objective (validation, downstream, OOD, transfer) Validation and downstream evaluation Low for validation, high for full evaluation Validation loss; generalization gap; downstream score
Compact cross-matrix between method families and effect objectives. ``++'' denotes a strong favorable prior, ``+'' a conditional or protocol-sensitive favorable effect, ``0'' protocol-dependent neutrality, ``-'' a conditional cost, and ``--'' a strong likely cost.
Family O1 O2 O3 O4 O5 O6
T1 Element-wise adaptive moment and scalar control ++ 0 -- ++ ++ +
T2 Matrix-level structural methods ++ -- - ++ + +
T3 Discretized directions + ++ ++ + 0 0
T4 State compression - + ++ - 0 -
T5 Geometry regularization + -- - ++ ++ ++
Optimizer instantiations under the four-axis decomposition (Axes I–IV of Section 3.2), spanning all five families (T1–T5). Axis III shows two faces of the same direction operator Φt: the LMO constraint ball and the preconditioner Ht. ℝd means all parameters are flattened into d independent scalar coordinates and updated element-wise, while ℝm×n means the update operates on a whole weight matrix and couples its entries. Pt denotes a low-rank projection onto the current subspace (learned for GaLore/Fira, random for APOLLO, column-orthogonal for Conda) with back-projection Pt⊤; a bar (e.g. v̄t) marks a quantity formed inside Pt. VR marks a variance-reduced (MARS) estimate, and matrix routing means only matrix parameters receive the matrix update while vector-like parameters follow an AdamW-style branch.
Optimizer Axis I: Domain Axis II: State estimator Axis III (LMO) Axis III (Precondition) Axis IV: Finalization
T1: Element-wise adaptive moment and scalar control
SGDM ℝd mt ℓ2 ball Ht=I LR
Adam, AdamW ℝd mt,vt adaptive ℓ∞ Ht=diag(vt) LR + decoupled WD
NAdam ℝd Nesterov mt, vt adaptive ℓ∞ Ht=diag(vt) Nesterov + LR + WD
AdaBelief ℝd mt,st adaptive ℓ∞ Ht=diag(st) LR + WD
ADOPT ℝd ordered, delayed mt,vt adaptive ℓ∞ Ht=diag(vt) LR + WD
Adan ℝd mt + grad-diff state adaptive ℓ∞ Ht=diag(vt) LR + WD
AdEMAMix ℝd short/long EMA adaptive ℓ∞ Ht=diag(vt) LR + WD
MARS-AdamW ℝd ct,mt,vt=EMA(ct2) adaptive ℓ∞ (VR) Ht=diag(vt) LR + decoupled WD
RAdam ℝd rectified mt,vt adaptive ℓ∞ Ht=diag(vt) LR + WD
Prodigy ℝd mt,vt + LR est. dt adaptive ℓ∞ Ht=diag(vt) automatic LR + WD
T2: Matrix-level structural methods
Muon ℝm×n Mt spectral (polar) Ht=MtMt⊤ LR + matrix routing
MARS-Shampoo ℝm×n (Lt,Rt) ct,mt,Lt,Rt metric ball (VR) Ht=Lt1/4⊗Rt1/4 LR + damping
Shampoo ℝm×n (Lt,Rt) mt,Lt,Rt metric ball Ht=Lt1/4⊗Rt1/4 LR + damping
SOAP QL,QR mt,vt,QL,QR adaptive ℓ∞ in QL,QR diag(vt) in QL,QR LR + WD
GaLore Pt m̄t,v̄t on Ptgt projected ℓ∞ H̄t=diag(v̄t) Pt⊤ back + LR + WD
Fira Pt (+res) m̄t,v̄t + residual projected ℓ∞ (+res) H̄t=diag(v̄t) Pt⊤ back + res + LR
RMNP ℝm×n (row) Mt row-normalized Ht=diag(MtMt⊤) LR + matrix routing
T3: Discretization and directional quantization
SignSGD ℝd gt fixed ℓ∞ Ht=diag(|gt|) LR
Lion ℝd mt fixed ℓ∞ Ht=diag(|mt|) LR + WD
MARS-Lion ℝd ct,mt fixed ℓ∞ (VR) Ht=diag(|mt|) LR + WD
T4: State compression and structural aggregation
AdaFactor ℝd (factored) row/col vt factors adaptive ℓ∞ factored diag(vt) LR + factored update
CAME ℝd (factored) factors + confidence adaptive ℓ∞ factored diag(vt) (+conf.) LR + factored update
Adam-mini ℝd (block) mt,vt (block) block ℓ∞ block-mean diag(vt) LR + WD
APOLLO Pt (rand.) m̄t,v̄t projected ℓ∞ H̄t=diag(v̄t) Pt⊤ back + LR
8-bit Adam ℝd (INT8) mt,vt (INT8) adaptive ℓ∞ Ht=diag(vt) in INT8 dequant + LR + WD
Conda Pt (col) vt (col) projected ℓ∞ col-wise diag(vt) Pt⊤ back + LR + WD
T5: Curvature-aware and geometric regularization
Sophia ℝd mt,ht clipped local Ht=ht LR + WD
AdaHessian ℝd mt,ht (Hutch.) metric ball Ht=ht LR + WD
AdamP ℝd mt,vt adaptive ℓ∞ Ht=diag(vt) radial projection + LR + WD
LAMB ℝd mt,vt adaptive ℓ∞ Ht=diag(vt) trust ratio + LR + WD
Benchmark evidence

Cross-scenario evidence, not one-setting ranking.

The webpage foregrounds the evidence used by the paper's optimizer-choice argument: Stage-1 C4 quality/runtime/memory trade-offs, Stage-2 FineWeb-Edu long-context transfer, and auxiliary O4/O5 stability and robustness probes.

Stage-1 Pareto frontiers for 1B models comparing perplexity with runtime and optimizer memory
Figure 9. Stage-1 1B Pareto frontiers: lower-left is better for PPL versus runtime and optimizer-state memory.
Quality frontier

APOLLO, Muon, MARS-Shampoo, and RMNP are competitive on short-context C4, but they occupy different cost regions.

Runtime frontier

Lion and AdamW are cheap; RMNP is the practical matrix-structured exception close to the efficient frontier.

Memory frontier

AdaFactor, APOLLO, and GaLore reduce optimizer state, but memory wins do not automatically transfer to harder regimes.

Transfer frontier

FineWeb-Edu 32k turns optimizer choice into a cross-architecture stability question rather than an absolute PPL ranking.

Optimizer-level heatmap of C4 perplexity runtime and memory for 24 optimizers
Figure 10. Stage-1 optimizer-level heatmap of C4 PPL, runtime, and memory. It shows why the short-context screen is multi-objective rather than a flat leaderboard.
Rank stability across optimizers and architectures in FineWeb-Edu long-context scenarios
Figure 11. Stage-2 cross-scenario rank stability under FineWeb-Edu 32k. SOAP transfers most consistently; APOLLO's short-context strength does not survive this setting.
Findings

No optimizer dominates every objective frontier.

Structured-matrix methods transfer stably but can be expensive; state-compressed methods can win memory under short contexts but degrade as input complexity grows; rankings cross systematically across domains.

SOAP

Strongest long-context cross-scenario quality, but its runtime and optimizer-state memory make it a quality ceiling, not a default.

RMNP / Muon

Matrix geometry is useful but architecture-aware: RMNP is the balanced option, while Muon is mechanistically interpretable.

APOLLO / AdaFactor

State compression is attractive under memory pressure, but APOLLO's short-context win collapses under long context.

AdamW / Lion

AdamW remains the stable reference anchor; Lion is cheap for exploration but carries an expected quality gap.

Family-level objective summary over six optimizer objectives
Figure 14. Family-level objective summary over O1-O6, connecting measured quality, runtime, memory, stability, robustness, and generalization back to the optimizer-family taxonomy.
Gradient-norm stability heatmap across optimizers and architectures
Figure 12. Auxiliary O4 stability analysis from gradient-norm dynamics: smooth training, final PPL, and transfer are related but distinct objectives.
Learning-rate perturbation robustness panels for optimizer sensitivity
Figure 13. O5 learning-rate perturbation robustness: tuned quality can hide sharp sensitivity when the learning rate is misspecified.
Mechanistic ablation of Muon on C4 350M showing core operations gain operations and ordering constraints
Figure 15. Mechanistic ablation of Muon on C4 350M. The full-width placement keeps the core recovery, gain design, and operator-order blocks readable.
Cross-scale/architecture validation of Muon's gain operations. Standard Muon, symmetric two-way LR scaling, post-NS Nesterov, and their combination; lower PPL is better. Gains stack on the standard Transformer but not on Gated DeltaNet.
Scenario Standard Muon Symmetric LR Scaling Post-NS Nesterov Both combined Best config.
Standard Transformer: gains are stackable
C4-LLaMA, 350M 16.60 16.52 16.57 16.51 Both combined
C4-LLaMA, 1B 13.72 13.64 13.64 13.58 Both combined
Linear attention: stacking effect disappears
FineWeb-Edu-32k, GDN-340M 24.34 24.02 24.12 24.12 Symmetric LR Scaling
Decision guide

Choose the optimizer by the binding constraint.

The paper's conclusion is not a single global winner. It is a constraint-matched decision rule: start from AdamW, then move only when quality, runtime, memory, stability, or cross-scenario transfer demands a different mechanism.

Reference anchor

AdamW

Use as the default baseline for general-purpose LLM pretraining: stable, inexpensive, interpretable, and the reference every other optimizer should beat.

Quality-efficiency balance

RMNP

Best practical alternative when a matrix-structured method is needed without the prohibitive runtime and memory cost of heavier preconditioners.

Quality ceiling

SOAP

Strongest long-context cross-architecture quality profile, useful when final quality dominates and compute/memory are not the bottleneck.

Mechanism analysis

Muon

Strong and transparent matrix-structured optimizer, but its behavior is topology-dependent and should be validated on the target architecture.

Memory pressure

AdaFactor / APOLLO

AdaFactor is the safer low-memory baseline. APOLLO is high reward but high risk: strong at short context, weak under long-context transfer.

Exploration budget

Lion

Cheap exploratory option with low per-step overhead, but the paper treats the quality gap as expected rather than incidental.

Tiered classification of the benchmarked optimizers.
Tier Optimizers
Tier I Muon, RMNP, AdamW
Tier II MARS-Lion, MARS-Shampoo, APOLLO, Conda, AdamP, MARS-AdamW, SOAP, Adan, Lion
Tier III RAdam, NAdam, Prodigy, AdaBelief, GaLore, Shampoo, 8-bit Adam, CAME, AdaFactor, Adam-mini, LAMB, Sophia
Full evidence tables

Central benchmark tables are preserved in full.

Wide paper tables are rendered as responsive HTML tables with all source rows and columns. On narrow screens, rows turn into labeled cards instead of requiring horizontal scroll.

Stage-1 screening on C4 (LLaMA, seq.\ 256). C4 validation PPL, optimizer-state memory (Mem), and per-step runtime (T) at four scales; lower is better. Grouped by family, sorted by 1B PPL.
Optimizer Venue 60M PPL 60M Mem GB 60M T ms 130M PPL 130M Mem GB 130M T ms 350M PPL 350M Mem GB 350M T ms 1B PPL 1B Mem GB 1B T ms
T1: Element-wise adaptive moment and scalar control
Adan TPAMI'24 30.25 0.433 2.32 22.84 1.000 4.72 17.29 2.742 12.06 14.35 9.977 39.67
RAdam ICLR'20 30.12 0.217 1.53 23.22 0.500 3.07 17.34 1.371 7.64 14.47 4.989 23.79
AdamW ICLR'19 30.08 0.217 1.14 23.18 0.500 2.31 17.78 1.371 5.97 14.48 4.989 18.62
NAdam ICLR'16 33.72 0.217 3.45 24.51 0.500 4.93 17.90 1.371 9.96 14.67 4.989 20.91
MARS-AdamW ICML'25 30.01 0.325 7.62 22.86 0.750 11.05 16.95 2.057 22.12 14.90 7.483 34.70
Prodigy ICML'24 33.44 0.433 8.36 24.13 1.000 12.29 18.27 2.742 24.30 15.61 9.977 36.78
AdaBelief NeurIPS'20 30.08 0.433 5.76 23.45 1.000 8.55 17.61 2.742 19.10 16.79 9.977 55.48
T2: Matrix-level structural methods
MARS-Shampoo ICML'25 30.03 0.325 26.27 22.56 0.750 37.94 16.82 2.057 78.71 13.72 7.483 513.7
Muon arXiv'25 28.26 0.109 21.01 21.81 0.250 30.48 16.60 0.686 61.66 13.72 2.495 379.0
RMNP ICML'26 29.88 0.109 3.26 22.54 0.250 4.63 16.85 0.686 9.32 13.87 2.495 16.94
SOAP ICLR'25 29.47 0.731 50.58 22.67 2.214 110.4 17.14 7.465 302.5 14.04 29.299 1371.5
GaLore ICML'24 34.56 0.062 4.21 25.32 0.199 5.88 19.18 0.426 11.85 14.29 0.790 15.29
Shampoo ICML'18 30.22 0.217 22.36 22.56 0.500 33.27 17.03 1.371 66.05 14.29 4.989 389.4
T3: Discretization and directional quantization
MARS-Lion ICML'25 32.41 0.325 5.72 25.68 0.750 8.49 18.78 2.057 17.11 15.73 7.483 24.77
Lion NeurIPS'23 35.94 0.109 2.07 25.56 0.250 3.01 19.30 0.686 5.80 17.02 2.494 12.48
T4: State compression and structural aggregation
APOLLO MLSys'25 30.86 0.062 8.62 22.74 0.149 12.65 16.43 0.426 26.21 13.53 0.790 28.65
Conda arXiv'25 28.65 0.245 4.88 21.91 0.595 7.11 16.45 1.703 13.90 14.25 6.317 62.33
8-bit Adam ICLR'22 30.46 0.110 4.11 23.30 0.254 7.27 17.67 0.697 16.89 14.53 2.534 42.38
CAME ACL'23 31.40 0.218 14.99 23.79 0.502 21.76 17.60 1.376 44.89 14.53 4.997 87.46
AdaFactor ICML'18 30.00 0.001 9.90 22.94 0.002 14.63 17.85 0.003 29.70 14.92 0.004 56.46
Adam-mini ICLR'25 30.50 0.109 5.68 23.62 0.251 8.31 18.12 0.686 16.68 15.51 2.495 20.81
T5: Curvature-aware and geometric regularization
AdamP ICLR'21 30.21 0.217 12.82 23.07 0.500 19.13 17.39 1.371 39.98 14.57 4.989 64.69
LAMB ICLR'20 30.03 0.217 9.14 23.40 0.500 13.17 17.25 1.371 26.62 16.09 4.989 44.18
Sophia ICLR'24 36.27 0.217 3.92 25.76 0.500 5.66 18.86 1.371 11.06 16.45 4.989 20.05

Full Stage-1 C4 screening table: all optimizer families, four scales, and PPL/memory/runtime values are retained.

Stage-2 cross-architecture generalization (FineWeb-Edu, 32k). Left block: WikiText test PPL (lower is better) at 340M and 1B across four architectures; absolute PPL is not comparable across architectures, so rows are ordered by PPL-based cross-scenario stability. Right block: downstream commonsense-reasoning accuracy (CS Avg, %, average over ten lm-eval-harness tasks; higher is better). Gray row = AdamW. The best PPL is almost always SOAP, whereas the best CS Avg is spread across SOAP, MARS-AdamW, RMNP, and Muon.
Optimizer PPL Tr++ 340M PPL Tr++ 1B PPL GDN 340M PPL GDN 1B PPL Delta 340M PPL Delta 1B PPL GLA 340M PPL GLA 1B CS Avg Tr++ 340M CS Avg Tr++ 1B CS Avg GDN 340M CS Avg GDN 1B CS Avg Delta 340M CS Avg Delta 1B CS Avg GLA 340M CS Avg GLA 1B
T1: Element-wise adaptive moment and scalar control
MARS-AdamW 24.57 18.94 24.17 20.04 26.79 20.67 28.28 21.89 52.50 57.46 54.91 58.18 51.69 56.80 51.24 55.71
AdamW 24.62 18.90 24.47 20.33 27.16 20.66 28.67 22.06 52.28 56.55 53.67 57.01 51.74 55.56 51.06 56.69
Adan 25.55 19.41 24.78 20.55 27.28 20.88 29.00 22.51 52.48 57.21 52.83 57.93 51.78 56.50 51.01 55.07
T2: Matrix-level structural methods
SOAP 23.90 18.72 23.85 19.86 26.02 20.38 27.04 20.62 53.75 57.71 54.77 57.22 52.60 56.49 52.21 57.57
RMNP 24.37 19.40 23.65 20.26 26.80 21.06 28.60 22.23 53.35 57.12 54.45 57.30 53.25 57.32 50.72 55.79
Muon 25.05 19.86 24.34 20.32 27.18 21.18 27.47 21.54 53.25 56.36 54.45 57.20 52.00 56.65 52.50 56.85
MARS-Shampoo 26.43 19.74 25.99 24.87 28.26 21.25 29.20 21.53 51.96 57.30 53.37 57.58 51.43 56.74 52.01 57.33
T3: Discretization and directional quantization
Lion 26.02 20.26 24.76 20.38 28.20 21.44 29.47 22.40 51.07 55.22 53.24 55.74 49.96 54.22 50.14 53.96
MARS-Lion 26.20 21.17 25.24 22.20 28.25 22.72 29.67 23.79 51.61 54.51 52.96 55.50 50.94 53.65 50.69 53.91
T4: State compression and structural aggregation
Conda 28.30 19.86 26.11 21.07 29.09 21.75 37.38 22.89 51.61 57.24 53.45 57.18 51.46 56.10 48.28 54.95
APOLLO 34.08 25.29 30.36 29.29 34.73 25.58 37.75 27.78 48.19 53.61 50.92 53.73 49.04 53.88 48.38 52.33
T5: Curvature-aware and geometric regularization
AdamP 24.68 19.04 24.32 20.29 26.77 20.68 28.66 21.86 51.69 56.82 53.82 57.07 51.53 56.73 51.14 55.31

Full Stage-2 long-context cross-architecture table with WikiText PPL and downstream commonsense-reasoning accuracy (CS Avg) across all eight scenarios.

Citation

BibTeX

@article{li2025omniopt,
  title = {{OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers}},
  author = {Siyuan Li and Jiabao Pan and Yumou Liu and Zhuoli Ouyang and Xin Jin and Xinglong Xu and Jingxuan Wei and Shengye Pang and Jintao Chen and Xuanhe Zhou and Conghui He and Cheng Tan},
	  year = {2025}
	}
Figure preview
Expanded selected paper figure