Meta-Learning Benchmark

A Meta-Learning Benchmark is an ML benchmarking task, which provides a standardized evaluation setting to assess a metalearning system’s or metalearning algorithm’s performance on a specific metalearning task.

• AKA: Few-shot Learning Benchmark, Cross-Task Benchmark.
• Context:
  • Task Input: A distribution of train/test tasks with defined support and query sets.
    • Optional Input: Task-specific configurations, augmentation settings.
  • Task Output: Adapted learner performance on unseen test tasks.
  • Performance Measure: Few-shot accuracy, test error, generalization gap, speed of adaptation.
  • Benchmark dataset(s): MiniImageNet, Omniglot, Meta-Dataset, tieredImageNet, FewRel.
  • It can compare metalearning approaches across diverse domains like vision, NLP, and RL.
  • It can include N-way K-shot protocols with reproducible splits.
  • It can support robustness testing across out-of-distribution or noisy task settings.
  • ...
• Example(s):
  • MiniImageNet, widely used for few-shot classification evaluations.
  • Omniglot Benchmark, for handwritten character recognition in low-resource settings.
  • Meta-Dataset, a diverse benchmark across multiple image domains.
  • FewRel Benchmark, for few-shot relation classification in NLP.
  • ...
• Counter-Example(s):
  • MNIST Benchmark, which evaluates a single-task classification setting.
  • ImageNet Benchmark, which does not involve few-shot or cross-task settings.
  • ...
• See: Metalearning Task, Metalearning Algorithm, Few-shot learning benchmark, Omniglot Benchmark, MiniImageNet.

---