Predictable Emergence: An Empirical Analysis of Whether Sharp Capability Jumps Follow from Smooth Per-Token Scaling Laws

Neural scaling laws describe the pretraining loss of a transformer as a smooth power law in model size, data, and compute, yet the capabilities practitioners care about are often reported to emerge abruptly with scale. Whether these sharp jumps are a genuine property of learning or an artifact of discontinuous metrics remains the most consequential open question for forecasting model behavior. We give a narrow, fully controlled, inference-only test of one falsifiable version of this question: are the location and shape of an “emergent” jump in exact-match accuracy quantitatively predictable, ahead of time, from the smooth power-law improvement of per-token accuracy? We study multi-digit integer addition—the canonical emergence example—because the answer length 𝐿 is an exactly controllable knob, and we evaluate the fully open Pythia suite (70M–12B), its deduplicated variant, and OPT (125M–6.7B), using released checkpoints only and no training. Per-token digit accuracy rises smoothly and monotonically with scale (0.47→0.91 for Pythia, a clean power law in the per-token error with exponent 𝛼 ≈ 0.36), while exactmatch accuracy shows the textbook family of sharp, length-dependent “emergences.” A one-parameter compositional model, EM ≈ 𝑝 𝐿eff with a single fitted per-position error correlation 𝜌 = 0.17, reproduces the entire 6 × 8 grid of exact-match curves with 𝑅 2 = 0.999 (median absolute error 0.004); the naive independent-token model (𝜌=0) already reaches 𝑅 2 = 0.98 but systematically under-predicts, exposing weak positive correlation as the only correction needed. All lengths, sizes, and model families collapse onto a single master curve EM = 𝑒 −𝑢 under the rescaled coordinate 𝑢 = 𝐿eff · (− ln 𝑝) (collapse RMSE ≤ 0.013). Fitting the per-token law on the four smallest models (≤1B) alone forecasts the held-out jumps of the larger models to within a mean absolute error of 0.017 in exact-match and 0.01 dex in threshold location. Continuous metrics (per-token accuracy, Brier score, edit distance) show no jump and no length dependence. We find no evidence of super-compositional behavior that a genuine, metric-invariant emergence would require—for arithmetic, within this scale range. Emergence here is not a new capability appearing at a threshold but a smooth per-token law imaged through a length-𝐿 nonlinearity, and it is predictable in advance.