2023 GenAIPassestheMPREWithNoPriorEt
Jump to navigation
Jump to search
- (Wilkins, 2023) ⇒ Stephanie Wilkins. (2023). “Gen AI Passes the MPRE With No Prior Ethics Training.”
Subject Headings: MPRE, Law-Supporting AI.
Notes
- The article from Legaltech News, dated November 16, 2023, discusses a study conducted by LegalOn Technologies where generative AI large language models (LLMs) like OpenAI’s GPT-4 and Anthropic’s Claude 2 were tested against the [[Multistate Professional Responsibility Exam (MPRE)[[, which assesses knowledge of legal ethics and is a requirement for bar admission in most U.S. jurisdictions.
- LLMs Performance on MPRE: GPT-4 and Claude 2 successfully passed the MPRE, showcasing the potential of AI in assisting legal professionals. GPT-4, in particular, outperformed the average human test-taker, scoring 74% which is 6 percentage points higher than the human mean. Claude 2's score was slightly above the threshold for passing, while other models like GPT-3.5 and PaLM 2 Bison scored below the passing threshold.
- Areas of Strength and Weakness: The LLMs showed varying performance in different subject areas. For example, GPT-4 performed well in conflicts of interest and client relationships but scored lower in areas like the safekeeping of funds.
- Implications for Legal Practice: The study underscores the capability of AI to support ethical decision-making in legal practices. However, it’s emphasized that AI should not replace the ethical judgment of legal professionals. Daniel Lewis, CEO of LegalOn, mentioned the need for further training for LLMs to guide ethical behavior.
- Opportunities for Enhancement: Gabor Melli, VP of AI at LegalOn, suggested that the results indicate opportunities to enhance AI performance with additional training, especially in more esoteric areas.
- Study Methodology: The study involved testing the LLMs against 100 simulated exams comprising 500 questions on legal ethics, created by Professor Dru Stevenson. The zero-shot approach meant the models received no prior specific training on legal ethics.
- Future Potential: LegalOn sees this as just the beginning, indicating that these results represent the floor, not the ceiling, of AI performance in legal ethics. The possibility of integrating more domain-specific knowledge into AI tools is also discussed.
- LegalOn’s Role in Legal AI: LegalOn, founded in Tokyo in 2017 and expanded to the U.S. in 2023, has been at the forefront of integrating AI into legal practices. Their products include AI Revise and LegalOn Templates, aimed at enhancing contract review and drafting processes.
Cited By
Quotes
Abstract
OpenAI’s GPT-4 and Anthropic’s Claude 2 passed the ethics exam, though researchers from LegalOn Technologies were quick to note that, while the technology has the potential to assist lawyers with ethical decision-making, it should not replace them in that role.
Body
- “These large language models really pick up on that content that’s written a lot about, and other more esoteric areas are going to be where it falters a bit,” Gabor Melli, VP of AI at LegalOn, observed to Legaltech News. In his opinion, these results “highlighted opportunities for enhancement” if additional training is provided to the LLMs, he added.
[[File:Comparison-of-Selected-LLM-Models-on-the-Legal-Ethics-Exam-767x633.jpg|alt=2023_GenAIPassestheMPREWithNoPriorEt.
References
;
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2023 GenAIPassestheMPREWithNoPriorEt | Stephanie Wilkins | Gen AI Passes the MPRE With No Prior Ethics Training | 2023 |