2023 AreEmergentAbilitiesofLargeLang
- (Schaeffer et al., 2023) ⇒ Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. (2023). “Are Emergent Abilities of Large Language Models a Mirage?.” doi:10.48550/arXiv.2304.15004
Subject Headings: LLM Emergent Behavior.
Notes
- It proposes a new perspective on emergent abilities in large language models (LLMs), arguing that these abilities may not be inherent to the models but rather a consequence of the metrics chosen for their evaluation.
- It offers an alternative explanation for emergent phenomena, suggesting the role of nonlinear or discontinuous metrics in creating the illusion of sudden capabilities as models scale up.
- It utilizes the InstructGPT/GPT-3 models in empirical analysis, demonstrating that switching from nonlinear or discontinuous metrics to linear or continuous ones unveils a smooth, predictable performance curve, challenging existing beliefs about emergent abilities.
- It conducts a comprehensive meta-analysis, scrutinizing claims of emergent abilities across a spectrum of tasks and metrics to reveal that these phenomena primarily emerge under specific, often nonlinear or discontinuous, evaluation metrics.
- It demonstrates through vision task experiments how changing metrics can induce the appearance of emergent abilities in various model architectures, emphasizing the profound impact of metric selection on perceived model capabilities.
- It highlights the significant influence of metric choice on the perceived performance of AI models, suggesting that inappropriate metric selection can lead to false perceptions of sudden emergent abilities as models scale.
- It advocates for a reevaluation of current evaluation practices within AI research, stressing the importance of selecting metrics that truly capture the gradual improvements in model performance to avoid misleading conclusions.
- It encourages further research into developing new evaluation metrics that can more accurately reflect the nuanced improvements of AI models over time and investigate the true nature of emergent phenomena.
- It contributes to the ongoing dialogue around AI model evaluation and scaling, pushing for a deeper understanding of model advancements and the critical role of thoughtful, rigorous metric selection in research.
Cited By
Quotes
Abstract
Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance. We present our alternative explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT / GPT-3 family on tasks with claimed emergent abilities; (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show to choose metrics to produce never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep networks. Via all three analyses, we provide evidence that alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.
References
;
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2023 AreEmergentAbilitiesofLargeLang | Rylan Schaeffer Brando Miranda Sanmi Koyejo | Are Emergent Abilities of Large Language Models a Mirage? | 10.48550/arXiv.2304.15004 | 2023 |