Natural Language Processing (NLP) System Benchmark Task

A Natural Language Processing (NLP) System Benchmark Task is a Computer Software Benchmarking Task that evaluates the performance of NLP systems.

• Context:
  • Task Input: NLP Dataset,
  • Task Output: NLP System's processed data (e.g. translated, POS tagged data).
  • Task Requirement(s):
    • Benchmark Datasets,
    • Benchmark Performance Metrics,
    • a ML Training System (optional),
    • a NLP Baseline System,
    • NLP Competing System (s).
  • It can (typically) be supported by an NLP Benchmark Corpus.
  • It can (often) be an input to an NLP System Evaluation Task.
• Example(s):
  • a Syntactic Parsing Benchmark Task such as: Penn Treebank Project;
  • a Semantic Relation Mention Recognition Benchmark Task such as: PPLRE Project or CPROD1 Task;
  • a Named Entity Recognition Benchmark Task;
  • a Coreference Resolution Benchmark Task;
  • a Question Answering Benchmark Task;
  • a Document Summarization Benchmark Task such as: DUC-2005 summarization.
  • a CoNLL Task.
  • a Natural Language Understanding (NLU) Benchmark Task such as:
    • a GLUE Benchmark (Wang et al., 2018),
    • a SuperGLUE Benchmark (Wang et al., 2019),
    • a RTE Challenge (Bentivogli et al., 2017),
  • a Linguistic Semantic Analysis Benchmark Task such as:
    • Semantic Textual Similarity Benchmark (STS-B) Task.
  • …
• Counter-Example(s):
  • a DBMS Benchmark Task.
  • a Visual Understanding Benchmark Task.
  • a Benchmark SAT Task.
• See: Natural Language Processing System, NLP Benchmark Diagnostic Dataset, Natural Language Translation Task, Natural Language Inference System, Lexical Entailment, Syntactic Parsing System, Morphological Analysis System, Word Sense Disambiguation, SemEval Task, LRE Map.

---

---