Law professors rate AI answers higher than peer responses in study
6 reported
A new study found that U.S. law professors rated answers from large language models (LLMs) higher than those written by their peers in a blinded evaluation of short-answer tutoring in contracts courses. Sixteen law professors created 40 representative questions, wrote answers, and judged 2,918 anonymized comparisons between human and LLM responses. The professors gave LLMs an average win rate of 75.33%, with models performing similarly to the best instructor. LLM responses were flagged as harmful only 3.53% of the time, compared to 12.06% for professor-written answers. The study, by Alejandro Salinas and colleagues, was reported by Marginal Revolution and cited by Andrew Curran and John Chamberlain. The article also notes a separate study in the Journal of Economic Literature finding that AI tools can mass-produce academic finance papers nearly indistinguishable from human research.
What’s reported
The study involved 16 U.S. law professors in a blinded evaluation of short-answer tutoring in contracts courses.
Professors created 40 representative questions and judged 2,918 anonymized comparisons between human and LLM responses.
LLMs received an average win rate of 75.33% over peer answers.
LLM responses were flagged as harmful 3.53% of the time, versus 12.06% for professor answers.
The study is by Alejandro Salinas and colleagues, reported via Marginal Revolution, Andrew Curran, and John Chamberlain.
A separate study in the Journal of Economic Literature found AI tools can produce academic finance papers nearly indistinguishable from human research.
Key figures
Alejandro Salinas (researcher)
Andrew Curran (cited source)
John Chamberlain (cited source)
Sources: marginalrevolution.com
