Glossary term · · 3 min read
GradNorm
A model reads “The parcel hasn't arrived; I need it tomorrow” and learns two things: choosing a department and recognizing urgency. We score each skill with a loss, a number describing prediction error; lower is better. After a while, the department loss has fallen from 1 to 0.2, while the urgency loss remains at 1. Fixed coefficients on both losses do not respond to one task making progress while the other stands still.
GradNorm is a method for automatically adjusting these coefficients during joint learning. It checks relative loss reduction and the strength of signals that change the model's shared settings. It is therefore an adaptive form of Task Weighting, which controls each task's contribution to the learning objective. The algorithm is described by Chen et al., §3 and Algorithm 1.

Equal strength does not mean equal progress
A gradient describes how the loss responds to small changes in the model's settings. Its norm is a single number measuring the signal's magnitude, without its direction. GradNorm compares norms for the same selected part of the shared settings, usually the last shared layer of computations.
In our example, assume both tasks currently have a norm of 3, already including their loss coefficients. Their signals are equally large, but department selection has reduced its loss to 20% of the initial value, whereas urgency still has 100%. These proportions average 60%. Dividing each by that average gives 1/3 and 5/3: a larger number indicates slower relative progress.
With the balancing strength set to α = 1, GradNorm sets target norms of 3 × 1/3 = 1 for the department and 3 × 5/3 = 5 for urgency. It adjusts the coefficients to move the signals toward those targets; it does not instantly set the norms to the target values. After updating, it rescales the coefficients so their sum remains equal to the number of tasks.
These are our own invented numbers. The change concerns the tasks' influence on further learning, not confidence in an individual answer. The mechanism belongs to Multi-task Learning, where several tasks use a shared part of a model.
Change the balancing rule
The target follows the formula in §3.2, Equation 1:
is the average current norm (3), and is the task's remaining-loss proportion divided by the mean of those proportions. The parameter controls how strongly differences in progress change the targets. At α = 0, both targets are 3, despite different progress. In the demonstration, switch α, then swap the tasks' progress: does the target follow the task's name or its progress?
The experiment only calculates targets for a single moment of learning; it does not simulate model updates.
Where the rule can fail
A lack of progress may come from a task the model cannot learn. The rule may then keep increasing its share. The authors describe this problem in Supplement §7.2. Also, changing signal magnitudes does not remove conflicting directions; Gradient Surgery addresses those. The paper's experiments concern synthetic data and images; the parcel message is our example, not a result for LLMs.
The text and illustration were prepared with AI assistance.