Legal Document Corpus
(Redirected from legal corpus)
Jump to navigation
Jump to search
A Legal Document Corpus is a legal law-focused domain-specific document corpus of legal documents (such as contract agreements, court decisions, and statutory texts).
- AKA: Legal Corpus, Legal Text Corpus, Law Document Corpus, Legal Document Collection, Juridical Document Corpus, Legal Text Database.
- Context:
- It can typically support Legal NLP Tasks through legal language processing and legal text analysis.
- It can typically enable Legal Analytics Systems through legal pattern extraction and legal trend identification.
- It can typically facilitate Legal AI Solution Development through legal training data and legal model fine-tuning.
- It can typically provide Legal Research Support through legal precedent search and legal citation analysis.
- It can typically contain Legal Document Types including contract documents, court opinions, statutory provisions, and regulatory filings.
- ...
- It can often require Legal Expert Annotation through legal concept tagging and legal relation marking.
- It can often undergo Legal Text Preprocessing through legal citation extraction and legal entity recognition.
- It can often support Legal Prediction Models through case outcome prediction and contract risk assessment.
- It can often enable Legal Document Classification through legal topic modeling and legal category assignment.
- It can often facilitate Legal Information Extraction through legal clause identification and legal term extraction.
- ...
- It can range from being a Small Legal Document Corpus to being a Large Legal Document Corpus, depending on its legal corpus size.
- It can range from being an Unannotated Legal Document Corpus to being an Annotated Legal Document Corpus, depending on its legal annotation level.
- It can range from being a Real-World Legal Document Corpus to being a Synthetic Legal Document Corpus, depending on its legal data source.
- It can range from being a Single-Jurisdiction Legal Document Corpus to being a Multi-Jurisdiction Legal Document Corpus, depending on its legal geographic scope.
- It can range from being a Historical Legal Document Corpus to being a Contemporary Legal Document Corpus, depending on its legal temporal coverage.
- It can range from being a Public Legal Document Corpus to being a Proprietary Legal Document Corpus, depending on its legal data accessibility.
- It can range from being a Monolingual Legal Document Corpus to being a Multilingual Legal Document Corpus, depending on its legal language diversity.
- ...
- It can be developed through a Legal Corpus Creation Task using legal document collection and legal text curation.
- It can be managed through a Legal Corpus Management Task using legal data governance and legal quality control.
- It can be annotated through a Legal Annotation Task using legal tagging tools and legal annotation guidelines.
- It can be validated through a Legal Corpus Validation Task using legal accuracy metrics and legal completeness measures.
- ...
- Example(s):
- Contract-Focused Legal Document Corpuses, such as:
- Commercial Contract Corpuses, such as:
- ContractNLI by Koreeda & Manning (2021), for contract natural language inference.
- CUAD Dataset by Atticus (2021), for contract understanding and analysis.
- Contract Understanding Dataset for contract clause extraction.
- Government Contract Corpuses, such as:
- Commercial Contract Corpuses, such as:
- Court Decision Legal Document Corpuses, such as:
- Supreme Court Decision Corpuses, such as:
- Lower Court Decision Corpuses, such as:
- Statutory Legal Document Corpuses, such as:
- Multilingual Legal Document Corpuses, such as:
- Specialized Legal Document Corpuses, such as:
- ...
- Contract-Focused Legal Document Corpuses, such as:
- Counter-Example(s):
- Biomedical Document Corpus, which contains medical documents rather than legal documents.
- News Article Corpus, which contains journalistic content rather than legal text.
- Fiction Literature Corpus, which contains creative writing rather than legal documents.
- Legal Dictionary, which provides legal term definitions rather than legal document collections.
- Legal Knowledge Base, which contains structured legal knowledge rather than raw legal text.
- See: Legal NLP, Legal Document Management, Legal Analytics, Contract Analysis System, Legal Information Retrieval, Judicial Decision Support System, Legal AI Application, Law Practice Technology, Legal Benchmark Dataset, Computational Law.
References
2023
- (GitHub, 2023) ⇒ https://github.com/neelguha/legal-ml-datasets
- This is a collection of pointers to datasets/tasks/benchmarks pertaining to the intersection of machine learning and law.
This page is continually being updated. If I missed something, please contact me at (Neel Guha) nguha@stanford.edu and I'll add it!
- Task agnostic datasets: These datasets can be used for pretraining larger models. Alternatively, you cause them to construct artificial tasks.
- Caselaw Access Project(https://case.law/): all official, book-published United States case law.
- Legifrance(https://www.legifrance.gouv.fr/): a French legal publisher providing access to law codes and legal decisions. Requires scraping ([Paper](http://ceur-ws.org/Vol-2645/paper2.pdf)).
- US Supreme Court Database(http://scdb.wustl.edu/): information about every case decided by the US Supreme Court between 1791 and today.
- European Parliment Proceedings(https://www.statmt.org/europarl/): Parallel text of the proceedings of the European Parliment, collected in 11 languages.
- US Code(https://uscode.house.gov/download/download.shtml): downloadable version of the US Code in XML format
- Patent Litigation Docket Reports(https://www.uspto.gov/learning-and-resources/electronic-data-products/patent-litigation-docket-reports-data): detailed patent litigation data on over 80k unique district court cases
- Pile of Law(https://huggingface.co/datasets/pile-of-law/pile-of-law): a 256GB dataset of legal, administrative, and contractual texts.
- Open Australian Legal Corpus(https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus): The first and only multijurisdictional open corpus of Australian legislative and judicial documents.
- Benchmarks which combine multiple types of tasks
- LexGlue(https://huggingface.co/datasets/lex_glue): a GLUE inspired set of legal tasks
- LegalBench(https://github.com/HazyResearch/legalbench): a large language model benchmark for legal reasoning
- Judgement prediction: Training a model to predict the outcome of a case from various case specific features.
- European Court of Human Rights(https://archive.org/details/ECHR-ACL2019): 11.5k cases from ECHR's public database. [Paper](https://www.aclweb.org/anthology/P19-1424/).
- Document/contract annotation: Training a model to annotate sentences/clauses/sections in a contract (or other document) according to various criteria (e.g. unfairness, argument structure, etc).
- Detecting unfair clauses from online terms-of-service(http://155.185.228.137/claudette/ToS.zip): ~12k sentences from 50 terms-of-service agreements. [Paper](https://arxiv.org/pdf/1805.01217.pdf).
- Usable Privacy Project Data(https://usableprivacy.org/data): a collection of datasets for privacy policies, including OPP-115, APP-350, MAPS, and the ACL/COLING 2014 Dataset.
- Contract extraction dataset(http://nlp.cs.aueb.gr/software_and_datasets/CONTRACTS_ICAIL2017/index.html): 3,500 English contracts manually annotated with 11 different contract elements. [Paper](http://nlp.cs.aueb.gr/pubs/icail2017.pdf).
- EURLEX with EUROVOC annotations(http://nlp.cs.aueb.gr/software_and_datasets/EURLEX57K/index.html): 57k legilsative documents from the EU's public document database, annotated with concepts from EUROVOC. [Paper](https://www.aclweb.org/anthology/W19-2209/).
- Cornell eRulemaking Corpus(https://facultystaff.richmond.edu/~jpark/data/jpark_lrec18.zip): Collection of 731 user comments on the the Consumer Debt Collection Practices rule by the CFPB, with annotations containing information about argument structure. [Paper](https://facultystaff.richmond.edu/~jpark/papers/jpark_lrec18.pdf).
- German rental agreements (in English)(https://github.com/sebischair/Legal-Sentence-Classification-Datasets-and-Models): ~913 sentences from German rental agreements annotated by semantic type. [Paper](https://www.researchgate.net/publication/332171940_Classifying_Semantic_Types_of_Legal_Sentences_Portability_of_Machine_Learning_Models).
- Segmenting US court decision opinions into issue parts(https://github.com/jsavelka/us-dec-func-iss-sgm/blob/master/trade_secret_cases.json): 316 court decisions on cyber crime and trade secrets, manually segmented into 6 content based "types" (encompassing categories like "Introduction", "Dissent", or "Background"). [Paper](http://ebooks.iospress.nl/volumearticle/50840)
- ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts(https://arxiv.org/abs/2110.01799)
- This is a collection of pointers to datasets/tasks/benchmarks pertaining to the intersection of machine learning and law.
2022
- (Kalamkar et al., 2022) ⇒ P. Kalamkar, A. Tiwari, A. Agarwal, S. Karn... (2022). “[Corpus for automatic structuring of legal documents](https://arxiv.org)”. In: arXiv preprint arXiv.
- QUOTE: ... for processing and organizing legal documents. In this paper, we introduce a ... corpus for structuring legal documents. In particular, we introduce a corpus of legal judgment documents in ...
2022
- (Kapoor et al., 2022) ⇒ A. Kapoor, M. Dhawan, A. Goel, T.H. Arjun... (2022). “[HLDC: Hindi legal documents corpus](https://arxiv.org)”. In: arXiv preprint arXiv.
- QUOTE: ... growing corpus as we continue to add more legal documents to HLDC. We release the corpus ... As a use-case for applicability of the corpus for developing legal systems, we propose the ...
2021
- (Malik et al., 2021) ⇒ V. Malik, R. Sanjay, S.K. Nigam, K. Ghosh... (2021). “[ILDC for CJPE: Indian legal documents corpus for court judgment prediction and explanation](https://arxiv.org)”. In: arXiv preprint arXiv.
- QUOTE: ... In this paper, we introduce INDIAN LEGAL DOCUMENTS CORPUS (ILDC) intending to ... could assist in legal case judgment prediction in an explainable way. ILDC is a corpus of case ...
2021
- (Koreeda & Manning, 2021) ⇒ Yuta Koreeda, and Christopher D. Manning. (2021). “ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts.” In: Findings of the Association for Computational Linguistics: EMNLP 2021.
- ABSTRACT: Reviewing contracts is a time-consuming procedure that incurs large expenses to companies and social inequality to those who cannot afford it. In this work, we propose "document-level natural language inference (NLI) for contracts", a novel, real-world application of NLI that addresses such problems. In this task, a system is given a set of hypotheses (such as "Some obligations of Agreement may survive termination.") and a contract, and it is asked to classify whether each hypothesis is "entailed by", "contradicting to" or "not mentioned by" (neutral to) the contract as well as identifying "evidence" for the decision as spans in the contract. We annotated and release the largest corpus to date consisting of 607 annotated contracts. We then show that existing models fail badly on our task and introduce a strong baseline, which (1) models evidence identification as multi-label classification over spans instead of trying to predict start and end tokens, and (2) employs more sophisticated context segmentation for dealing with long documents. We also show that linguistic characteristics of contracts, such as negations by exceptions, are contributing to the difficulty of this task and that there is much room for improvement.
- QUOTE: Helped by their accessibility, there exist multiple prior works on legal NLI. One of the subtasks in COLIEE-2020 shared task (Rabelo et al., 2020) was, given a court decision Q and relevant cases, to extract relevant paragraphs from the cases and to classify whether those paragraphs entail “Q” or “not Q”. Holzenberger et al. (2020) introduced a dataset for predicting an entailment relationship between a statement and a statute excerpt. While they are both “legal” and “NLI”, statutes and contracts exhibit different characteristics including the fact that statutes/cases tend to be written in consistent vocabulary and styles. Moreover, there only exists a single right answer for a hypothesis in case/statute law NLI, whereas a hypothesis can be entailed by or contradicting to each contract in our task; i.e., hypotheses and documents have one-to-one relationships in case/statute law NLI, but they have many-to-many relationships in our task.
As discussed in Section 1, our task has practical and scientific significance compared to information extraction for contracts (Leivaditi et al., 2020; Hendrycks et al., 2021). We showed in our experiments that the NLI part of our task is much more challenging than the evidence identification task. Furthermore, we gave observations to linguistic characteristics of our dataset that are lacking in these prior works.
Lippi et al. (2019) presented a dataset where certain types of contract clauses are identified and annotated with “clearly fair”, “potentially unfair” or “clearly unfair”. While the format of the task input and output is quite similar, our task requires reasoning over a much diverse set of hypotheses than just fair or unfair. Similarly, fact extraction and claim verification tasks (Thorne et al., 2018; Jiang et al., 2020), where the task is to extract facts from Wikipedia articles and to classify whether the claim is entailed by the facts, have similar input and output formats. ...