An AI Chatbot That Knows What Your Course is About
In Short:
A custom course-specific AI chatbot can serve as a round-the-clock tutor
Restricting it to your own materials helps maintain consistency and relevant answers
Usage data shows students ask more questions and revise more frequently
Proper guardrails and training are essential to prevent misinformation and cheating
In theory, there’s a lot of good that an AI-savvy student can do. They can ask unlimited questions 24/7, they can dive deeper into and beyond the textbook while personalising their learning in a way that only the very rich could afford until recently (through an army of tutors). Yet most students aren’t tech-savvy, so how can teachers encourage, monitor, and safeguard AI usage?
In September 2024, my colleagues and I started an experiment where we created a custom AI chatbot that is programmed with data specific to an Intro to Psychology course at a community college in Hong Kong. This chatbot had access to the course’s lectures, outlines, and other data, and it is explicitly instructed to act as a course tutor only and refuse to answer any unrelated queries.
Students were introduced to the chatbot during course tutorials, which also came with very basic AI training such as showing them how to generate revision questions, to ask for analogies to aid understanding, to explain ideas in different languages, and of course telling them that AI answers can be wrong.
The start-of-semester survey gave us a few pleasant surprises: the majority of students know that AIs can give false answers, and many also note that cheating is a major risk (usually in the context of other students using it to cheat and creating unfair competition). However, they also reported that their AI use cases have been for replacing search engines (which is problematic because AIs can give factual mistakes), or using it to write/draft homework.
The chatbot’s usage rose gradually, with each midterm and deadline seeing a bump in usage. In the end-of-semester survey, AI usage frequency for the majority of students went from “twice a month” or “monthly or less” to “few times a week” and even “everyday”. We also received very different comments like “it’s easier to ask AI than to ask the teacher questions” or that “I can revise for a quiz faster with mock questions”. We believe that while rough around the edges, setting up AI course tutors represents one practical way to deploy AI while aligning with today’s curricula.
Several factors made this course well-suited to such a chatbot. It’s a large class, with nearly 300 students, which meant limited direct access to the lecturer, so there’s a lot to gain by introducing a 24/7 AI tutor. The course content is also introductory and well-established, such that even the base AI model (we used GPT-4o) could already give very good answers to the students’ questions, except the customisation ensures there is consistency and continuity in glossary, definitions, and assessment formats. Not to mention, it’s instructive to go through the chat transcripts to see whether students copied AI output verbatim, or to see how they are using AI to learn.
This project has been very encouraging and we are going deeper in the current semester, aiming to provide better training for students while scaling this tool to other courses by popular demand. I hope to give further positive updates as the semester progresses.
In the next edition…
How AI coding agents are used to help secondary students with zero coding experience go from concept to prototype within an hour, in a course that’s about using new technologies to promote sustainability and inclusion.
Send us your AI in Action questions, use cases, and ideas at info@goodfuture.foundation!
