Meta-Learning Algorithm

A Meta-Learning Algorithm is a machine learning algorithm that can be implemented by a meta-learning system to solve a meta-learning task by optimizing the process of learning across tasks.

• AKA: Learning-to-Learn Algorithm, Meta Optimizer.
• Context:
  • It can be classified as:
    • Optimization-based, e.g., Model-Agnostic Meta-Learning (MAML).
    • Metric-based, e.g., Prototypical Networks, Matching Networks.
    • Model-based, e.g., Memory-Augmented Neural Networks.
  • It can train base learners using nested optimization loops for generalization.
  • It can leverage gradient-based, memory-based, or embedding-based adaptations.
  • It can support continual, few-shot, and task-agnostic settings.
  • It can improve data efficiency and enable rapid learning on new tasks.
  • ...
• Example(s):
  • Model-Agnostic Meta-Learning, using inner and outer gradient loops.
  • Reptile, a simpler first-order variant of MAML.
  • SNAIL, combining temporal convolutions and attention for task adaptation.
  • LSTM-based Meta-Optimizer, which learns to update weights of a learner.
  • ...
• Counter-Example(s):
  • Stochastic Gradient Descent, which optimizes task-specific parameters but not meta-level adaptation.
  • Transfer learning heuristics, which adapt models post-training but do not optimize learning processes.
  • Stacked Learning Algorithm.
  • ...
• See: Meta-Learning Benchmark, Few-shot learning algorithm, Optimization algorithm, Memory-Augmented Neural Network,Metaheuristic, Weighted Majority Learning, Meta-Data, Ensemble Learning Algorithms, Inductive Bias.

---

---

References

2015

• (Wikipedia, 2015) ⇒ http://en.wikipedia.org/wiki/meta_learning_(computer_science) Retrieved:2015-6-16.
  • Meta learning is a subfield of Machine learning where automatic learning algorithms are applied on meta-data about machine learning experiments. Although different researchers hold different views as to what the term exactly means (see below), the main goal is to use such meta-data to understand how automatic learning can become flexible in solving different kinds of learning problems, hence to improve the performance of existing learning algorithms.

    Flexibility is very important because each learning algorithm is based on a set of assumptions about the data, its inductive bias. This means that it will only learn well if the bias matches the data in the learning problem. A learning algorithm may perform very well on one learning problem, but very badly on the next. From a non-expert point of view, this poses strong restrictions on the use of machine learning or data mining techniques, since the relationship between the learning problem (often some kind of database) and the effectiveness of different learning algorithms is not yet understood.

    By using different kinds of meta-data, like properties of the learning problem, algorithm properties (like performance measures), or patterns previously derived from the data, it is possible to select, alter or combine different learning algorithms to effectively solve a given learning problem. Critiques of meta learning approaches bear a strong resemblance to the critique of metaheuristic, which can be said to be a related problem.

2005

• (Cohen & Carvalho, 2005) ⇒ William W. Cohen, and Vitor R. Carvalho. (2005). “Stacked Sequential Learning.” In: Proceedings of the 19th international joint conference on Artificial intelligence.
  • QUOTE: We describe a new sequential learning scheme called “stacked sequential learning”. Stacked sequential learning is a meta-learning algorithm, in which an arbitrary base learner is augmented so as to make it aware of the labels of nearby examples.

---