Scale invariance
You double every number in part of a model. Does the answer change? Sometimes a following normalization removes the shared scale and the result stays the same. In other arrangements, this rescaling changes behavior.
Scale invariance means invariance under a specified rescaling. You must state what is scaled and what should remain unchanged: model output or its error value. In this note, we ask about multiplying weights by a positive number.
A simple example: lists [1, 2] and [2, 4], divided by their geometric lengths — the square root of the sum of squares — yield the same normalized direction. Normalization can therefore remove information about overall magnitude. This gives the direction of weight changes and their magnitude different roles in learning.
An entire Transformer cannot be considered invariant just because it contains RMSNorm. Other computational paths and final candidate scores can retain scale's influence. The study of the effective learning step, §2 and §5, distinguishes mathematical symmetry from empirical similarity of training curves.