Bilingual Evaluation Understudy (BLEU) Performance Measure

A Bilingual Evaluation Understudy (BLEU) Performance Measure is a precision-based n-gram-based text generation performance measure that evaluates generated texts against reference texts.

AKA: BLEU Metric, BLEU Evaluation Metric, Bilingual Evaluation Understudy.
Context:
- Metric Input: Generated Text Output, Reference Text Set, BLEU Configuration Parameters
  - Optional Input: BLEU N-gram Order, BLEU Brevity Penalty Parameter, BLEU Smoothing Method
- Metric Output: BLEU Score, BLEU Precision Value, BLEU Brevity Penalty, BLEU N-gram Precision
- Metric Performance Measure: BLEU Human Correlation Coefficients with human quality judgments
- ...
- It can typically calculate BLEU Modified Precision through BLEU clipped n-gram counting.
- It can typically apply BLEU Brevity Penalty through BLEU length ratio computation.
- It can typically compute BLEU Geometric Mean through BLEU n-gram precision averaging.
- It can typically evaluate BLEU Corpus-Level Scores through BLEU segment aggregation.
- ...
- It can often assess BLEU Text Fluency through BLEU higher-order n-grams.
- It can often measure BLEU Content Adequacy through BLEU unigram matching.
- It can often support BLEU Multi-Reference Evaluation through BLEU reference set comparison.
- It can often enable BLEU Statistical Significance Testing through BLEU bootstrap resampling.
- ...
- It can range from being a Document-Level BLEU Measure to being a Sentence-Level BLEU Measure, depending on its BLEU evaluation granularity.
- It can range from being a Case-Sensitive BLEU Measure to being a Case-Insensitive BLEU Measure, depending on its BLEU matching criteria.
- It can range from being a Tokenized BLEU Measure to being a Detokenized BLEU Measure, depending on its BLEU preprocessing method.
- It can range from being a Standard BLEU Measure to being a Smoothed BLEU Measure, depending on its BLEU zero-count handling.
- ...
- It can be implemented by a BLEU Evaluation System using BLEU scoring algorithms.
- It can be standardized through BLEU SacreBLEU Implementation for BLEU reproducible evaluation.
- It can be optimized in BLEU System Training through BLEU optimization objectives.
- It can be validated against BLEU Human Evaluation Studys using BLEU benchmark datasets.
- ...
Example(s):
- BLEU Standard Variants, such as:
  - BLEU-1, which measures BLEU unigram precision for BLEU word-level accuracy.
  - BLEU-2, which evaluates BLEU bigram precision for BLEU phrase-level fluency.
  - BLEU-3, which computes BLEU trigram precision for BLEU local coherence.
  - BLEU-4, which calculates BLEU 4-gram precision for BLEU standard evaluation.
- BLEU Modified Variants, such as:
  - iBLEU (Interactive BLEU), which supports BLEU interactive debugging and BLEU incremental scoring.
  - SacreBLEU, which provides BLEU standardized tokenization for BLEU reproducible results.
  - Self-BLEU, which measures BLEU text diversity in BLEU generation tasks.
  - NIST Metric, which modifies BLEU n-gram weighting based on BLEU information content.
- BLEU Application-Specific Usages, such as:
- the one initially proposed for machine translation in (Papineni et al., 2002).
- ...
Counter-Example(s):
- ROUGE Metric, which emphasizes recall-based evaluation rather than BLEU precision-based evaluation.
- METEOR Metric, which includes synonym matching and stemming unlike BLEU exact matching.
- BERTScore, which uses contextual embeddings instead of BLEU surface-level n-grams.
- CIDEr Metric, which incorporates TF-IDF weighting for image captioning evaluation.
- Perplexity Measure, which evaluates language model quality rather than BLEU generation quality.
See: Text Generation Evaluation, Machine Translation, Image Captioning, Dialogue Generation, Precision Metric, N-gram Matching, Automatic Evaluation.

References

2018

(Wikipedia, 2018) ⇒ https://en.wikipedia.org/wiki/BLEU Retrieved:2018-8-27.
- BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is" – this is the central idea behind BLEU. BLEU was one of the first metrics to claim a high correlation with human judgements of quality, and remains one of the most popular automated and inexpensive metrics.
  Scores are calculated for individual translated segments — generally sentences — by comparing them with a set of good quality reference translations. Those scores are then averaged over the whole corpus to reach an estimate of the translation's overall quality. Intelligibility or grammatical correctness are not taken into account.
  BLEU’s output is always a number between 0 and 1. This value indicates how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. Few human translations will attain a score of 1, since this would indicate that the candidate is identical to one of the reference translations. For this reason, it is not necessary to attain a score of 1. Because there are more opportunities to match, adding additional reference translations will increase the BLEU score.

Bilingual Evaluation Understudy (BLEU) Performance Measure

References

2018

2017

2011

2002

Navigation menu

Search