Natural Language Processing (NLP) System Benchmark Task

A Natural Language Processing (NLP) System Benchmark Task is a AI benchmarking task that evaluates the performance of NLP systems.

• Context:
  • Task Input: NLP Dataset,
  • Task Output: NLP System's processed data (e.g. translated, POS tagged data).
  • Task Requirement(s):
    • Benchmark Datasets,
    • Benchmark Performance Metrics,
    • a ML Training System (optional),
    • a NLP Baseline System,
    • NLP Competing System (s).
  • It can (typically) be supported by an NLP Benchmark Corpus.
  • It can (often) be an input to an NLP System Evaluation Task.
  • ...
• Example(s):
  • a Syntactic Parsing Benchmark Task such as: Penn Treebank Project;
  • a Semantic Relation Mention Recognition Benchmark Task such as: PPLRE Project or CPROD1 Task;
  • a Named Entity Recognition Benchmark Task;
  • a Coreference Resolution Benchmark Task;
  • a Question Answering Benchmark Task;
  • a Document Summarization Benchmark Task such as: DUC-2005 summarization.
  • a CoNLL Task.
  • a Natural Language Understanding (NLU) Benchmark Task such as:
    • a GLUE Benchmark (Wang et al., 2018),
    • a SuperGLUE Benchmark (Wang et al., 2019),
    • a RTE Challenge (Bentivogli et al., 2017),
  • a Linguistic Semantic Analysis Benchmark Task such as:
    • Semantic Textual Similarity Benchmark (STS-B) Task.
  • a Law NLP Benchmark Task.
  • …
• Counter-Example(s):
  • a DBMS Benchmark Task.
  • a Visual Understanding Benchmark Task.
  • a Benchmark SAT Task.
• See: Natural Language Processing System, NLP Benchmark Diagnostic Dataset, Natural Language Translation Task, Natural Language Inference System, Lexical Entailment, Syntactic Parsing System, Morphological Analysis System, Word Sense Disambiguation, SemEval Task, LRE Map.

---

---

References

2023

• (Wikipedia, 2023) ⇒ https://en.wikipedia.org/wiki/Natural-language_programming Retrieved:2023-11-12.
  • Natural-language programming (NLP) is an ontology-assisted way of programming in terms of natural-language sentences, e.g. English. A structured document with Content, sections and subsections for explanations of sentences forms a NLP document, which is actually a computer program. Natural language programming is not to be mixed up with natural language interfacing or voice control where a program is first written and then communicated with through natural language using an interface added on. In NLP the functionality of a program is organised only for the definition of the meaning of sentences. For instance, NLP can be used to represent all the knowledge of an autonomous robot. Having done so, its tasks can be scripted by its users so that the robot can execute them autonomously while keeping to prescribed rules of behaviour as determined by the robot's user. Such robots are called transparent robots as their reasoning is transparent to users and this develops trust in robots. Natural language use and natural-language user interfaces include Inform 7, a natural programming language for making interactive fiction, Shakespeare, an esoteric natural programming language in the style of the plays of William Shakespeare, and Wolfram Alpha, a computational knowledge engine, using natural-language input. Some methods for program synthesis are based on natural-language programming.

---