Syracuse University College of Law Professor Jack Graves has developed an Artificial Intelligence bot that uses curated, course-specific materials to assist students in mastering the applicable legal rules and their application. These digital “coaches” are available 24/7 to assist students in understanding challenging concepts and then to quiz students on the application of these concepts, providing immediate feedback in a variety of Q&A formats. Thus far, Graves has deployed the concept in his Evidence and Contracts courses.
Graves uses OpenAI’s private custom GPT feature, which allows him to provide students with an interactive experience that is narrowly tailored to Graves’s specific course. Graves accomplishes this with (1) a comprehensive set of instructions (i.e., prompts) telling the custom Coach exactly what to do and providing guardrails to keep it focused on the objective of assisting students in this course; and (2) uploaded copies of the course text and other key instructional materials that facilitate Retrieval Augmented Generation (RAG). This domain specific RAG layer increases the accuracy of the Coach’s responses in the context of this specific course and dramatically reduces the potential for errors, as compared to a generic LLM trained on generic data of varying quality.
This “walled garden” of course-specific material addresses the common issue with Large Language Model (LLM) AI platforms that indiscriminately draw from all information on the internet. “The LLMs pick up a good deal of erroneous information from unreliable sources, and they miss a lot of really good information that’s behind firewalls,” said Graves.
“The bot has been instructed to respond to students when they ask for answers by walking them through in a Socratic-style dialog much as I might in class or office hours,” said Graves. “When assisting students, the Coach relies first and foremost on the information uploaded in its RAG layer, not only helping to explain and quiz the students on accurate course doctrine but pointing students directly to appropriate sources within the course text itself.”
Of course, the key to this approach is a collaborative relationship between Graves and the publisher of his course textbooks. While Graves is a co-author of his Contracts textbook, the copyright is held by West Academic (the publisher of both the Learning Evidence and Learning Contracts textbooks used by Graves). Thus, Graves worked closely with West Academic in developing an approach that would appropriately protect all copyrighted material uploaded to the Coach’s RAG layer, and his use of both Learning Evidence and Learning Contracts is done under license from West Academic.
The use of the primary course text within the RAG layer effectively expands the value of the text far beyond the initially assigned readings. At the core of the Coach’s domain-specific content, the textbook continues to anchor the Coach’s role in assisting and quizzing students as they better learn to apply that content.
Graves is quick to explain, “The Coach does not replace basic course prep or attendance. It is purely a supplement to these traditional teaching and learning tools—albeit a very effective one, arguably far more effective than traditional generic study aids or generic LLMs often used by students today. Perhaps most valuable is the Coach’s ability to provide students with unlimited opportunities to apply the course material in a variety of assessment formats, all of which are subject to immediate feedback. At the end of the day, this is often the single most effective teaching and learning tool for law students, and the Coach provides this tool in a manner that is always available and fully aligned with course content and course outcomes. Graves emphasizes, “the teacher’s role is not being outsourced to the Coach—it is being supplemented in new ways for which narrowly tailored AI is uniquely suited.”
Students access the Coach through a dedicated course link, which provides for private interaction between student and Coach, unless the student voluntarily decides to share the unique link generated by a specific conversation. The initial privacy of the conversation encourages students to ask questions they might otherwise be uncomfortable raising (the proverbial “dumb question,” which is often anything but). However, it also allows students to use the Coach in collaborative study sessions or to forward a conversation to Graves for further exploration. This latter feature is particularly useful in terms of quality control of both student prompts and responses by the Coach.
Graves explains, “During the past two semesters, I’ve seen a few responses from the Coach that could be improved and one blatant error. However, the vast majority of interactive challenges arose from imperfect student prompts.” Thus, the students get two additional benefits from using the Coach: (1) they learn the importance of effective inputs (prompts); and (2) they learn the importance of verifying outputs. While the Evidence and Contract Coaches have proven very accurate (Graves directly tests them regularly himself, in addition to frequent student feedback), AI remains imperfect, and Graves has continued to “fine-tune” his Coaches by uploading additional course-specific material based on his own testing and observations of student/Coach interactions. And Graves further explains that his Coaches have thus far proven “far more dependable than generic LLMs alone or generic traditional study aids.”
Graves teaches exclusively in the College of Law’s JDinteractive hybrid online program, so the 24/7 availability of his Coaches is particularly important to a body of students located around the world. “This has allowed me to be more efficient and effective with my time while giving our global students a uniquely tailored experience that will help them master course material, while being available at any time that is convenient to them,” said Graves.
Graves also teaches the “AI and the Virtue of Law” residency.