Competition on Legal Information Extraction/Entailment (COLIEE)

A Competition on Legal Information Extraction/Entailment (COLIEE) is an NLP competition focused on extracting and reasoning with legal information.

• Context:
  • It can be organized annually.
  • It can feature tasks involving Legal Information Extraction and Legal Textual Entailment.
  • It can have participation open to teams worldwide.
  • It can use newly annotated Legal Document Data.
  • It can benchmark techniques for Legal AI.
  • …
• Example(s):
  • COLIEE 2018 - Held at JURIX 2018. Included citation prediction and summarization tasks.
  • COLIEE 2019 - Held at ICAIL 2019. Added QA and RTE tasks.
  • COLIEE 2021 - Held at JURIX 2021. Added legal judgment prediction task.
  • COLIEE 2022 [1] - held at JURISIN 2022.
  • COLIEE 2023 [2] - held at ICAIL 2023.
  • …
• Counter-Example(s):
  • GLUE - A more general NLP benchmark, not focused on legal texts.
  • SQuAD - A reading comprehension dataset, not specialized for law.
• See: Legal AI, Natural Legal Language Processing, Case Law.

---

---

References

2022

• (Rabelo et al., 2022) ⇒ J. Rabelo, Randy Goebel, M.Y. Kim, Y. Kano, .... (2022). "Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021." In: The Review of Contemporary Law, 45(2), 463-477. [doi:10.1007/s42241-021-00134-x]
  • ABSTRACT: We summarize the 8th Competition on Legal Information Extraction and Entailment. In this edition, the competition included five tasks on case law and statute law. The case law component includes an information retrieval Task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes an information retrieval Task (Task 3), an entailment/question answering task based on retrieved civil code statutes (Task 4) and an entailment/question answering task without retrieved civil code statutes (Task 5). ....

2019

• (Kano et al., 2019) ⇒ Y. Kano, M.Y. Kim, M. Yoshioka, Y. Lu, J. Rabelo, .... (2019). ["Coliee-2018: Evaluation of the Competition on Legal Information Extraction and Entailment."](https://link.springer.com/chapter/10.1007/978-3-030-32381-3_16) In: New Frontiers in Artificial Intelligence, JURIX 2019, 137–146. [doi:10.1007/978-3-030-32381-3_16]
  • QUOTE: "... the evaluation of the 5th Competition on Legal Information Extraction/Entailment 2018 ... of the COLIEE-2018 competition. For the case law, Task 1 retrieves noticed cases (information ..."

2018

• (Kano et al., 2018) ⇒ Yoshinobu Kano, Miyoung Kim, …, R. Goebel, and K. Satoh. (2018). "COLIEE-2018: Evaluation of the Competition on Legal Information Extraction and Entailment.” In: JSAI-isAI Workshops.

2016

• (Kim et al., 2016) ⇒ M.Y. Kim, Randy Goebel, Y. Kano, K. Satoh (2016). ["COLIEE-2016: Evaluation of the Competition on Legal Information Extraction and Entailment."](https://www.researchgate.net/publication/309559477_COLIEE-2016_Evaluation_of_the_Competition_on_Legal_Information_Extraction_and_Entailment) In: JURIX 2016.
  • QUOTE: "... competitions on legal information extraction/entailment (COLIEE 2014, COLIEE 2015) on a legal ... experimental effort in the legal information extraction/retrieval field. We held the third ..."

---