LexGLUE Benchmark

From GM-RKB
(Redirected from LexGLUE)
Jump to navigation Jump to search

A LexGLUE Benchmark is a legal text analysis benchmark that evaluates natural language understanding models specifically in the context of legal language.



References

2024

  • https://huggingface.co/datasets/coastalcph/lex_glue
    • NOTES:
      • LexGLUE is a benchmark dataset designed to evaluate the performance of NLP methods on legal tasks. It includes seven existing legal NLP datasets spanning multiple domains like European Court of Human Rights, US law, EU law, and contracts.
      • The tasks covered include multi-class classification, multi-label classification, and multiple choice question answering. This allows testing NLP models on a variety of legal language understanding challenges.
      • The goal is to develop generic legal language models that can perform well across tasks with limited fine-tuning, making it easier for NLP researchers to apply models to legal domains.
      • The current leaderboard includes results from Transformer-based pre-trained language models like BERT, RoBERTa, DeBERTa, and legal-domain adapted versions. The best models achieve micro-F1 scores in the high 70s to low 80s averaged across tasks.
      • Dataset sizes range from around 7,800 examples for SCOTUS to 80,000 examples for LEDGAR, with most containing training, development and test splits. All datasets are in English.
      • Example data fields include text excerpts like court opinions or contract clauses, classification labels like relevant laws or contract provision types, and multiple choice options for question answering.
      • While a valuable resource, more information is needed on annotation procedures, potential biases, and social impact considerations to responsibly leverage and expand the LexGLUE benchmark going forward.

2023

2021