April 29, 2022
Next week is the final “dead week” of classes. Thanks to everyone who attended the final CNS MTI workshop of the 2022 spring semester two Tuesdays ago led by Debbie Garrity and Dan Sloan on alternate learning assessment options. The panel (Tanya Dewy, Meena Balgopal, Kimberly Jeckel and Jen Todd) provided great examples and insights from their wealth of experience. The discussion that ensued was interesting and productive. Topics and ideas put forward included more frequent assessments (shorter weekly summative quizzes and four synthesis exams with questions higher on the Bloom’s scale), address cheating by having timed open book/notes exams, moving from bubble sheets to on-Canvas exams, graded problem sets on Canvas, grading using Grade Scope, moving toward wearing an educator hat rather than a police hat, naming assessments something other than “quizzes” and “exams” (“practice”, “demonstrate”, “put it all together” assessments). We also considered whether students learn better on paper or on a computer and inequities introduced when requiring access and use of a computer and the internet to complete assessments. I have attached a copy of the slides and included a link to a recording of the workshop. For me, improving how we assess learning in college courses represents the last frontier and I hope we can continue this discussion next fall.
Related to the topic of assessments, Beckie Supiano posted recent Chronicle of Higher Education April 28, 2022 “Teaching” article titled “The Case for Reminding Students about Assignments” provided below. I recall discussing in some committee the use of chatbots (automated application for text or text-to-speech conversations) for students during admissions to remind them and help them with filling out forms, financial aid, etc. I am not sure if it was implemented at CSU. Instructors at Georgia State University have been testing chatbots in conjunction with data analytics in courses to remind students to complete assignments with apparent success, particularly for historically excluded/underserved students.
Chatbots for the Classroom
I’ve long been fascinated by efforts to use behavioral “nudges”—low-cost, low-touch interventions to encourage, but not require, students to take a particular action. Much of the early research on nudging in higher ed focused on helping students apply for admission and financial aid — my Chronicle beat before I switched to teaching — and I covered it a bunch over the years.
That reporting culminated in this 2019 story, which tried to make sense of some disappointing findings from several high-profile efforts to expand on promising early studies.
One concern that seems to pop up in every discussion of nudging is that it could make college too easy for students. Shouldn’t they be able to navigate their way without text-message reminders? Most observers, I’ve found, can be convinced that helping students apply for financial aid is in-bounds: Wealthy students don’t have to file, and it’s hard to argue that a process that begins with finding your parents’ tax returns has much to do with someone’s ability to succeed academically.
The idea of bringing these interventions into the classroom, though, is more fraught.
So when I saw preliminary results from an academic-nudging experiment at Georgia State University, my interest was piqued. Georgia State is well known for its use of data analytics to support student success, and a chatbot it created to send students reminders and answer their questions was among the more-promising directions for nudging mentioned in my 2019 article.
The university is now testing chatbots in a classroom setting: an online section of a large, required government course. The first wave of results, based on 500 students who took the course in the fall, are outlined in a new working paper led by Katharine Meyer, a postdoctoral student at Brown University. The research team found that the chatbots had significant effects on student performance — especially among students from historically underrepresented groups.
Sixty percent of students in the experiment’s control group earned B’s or better in the course. The assistance of the chatbots increased the chances students earned such grades by eight percentage points. The researchers also looked at the impact on first-generation students, 45 percent of whom earned B’s or higher in the control group. Receiving the messages increased the chances first-gen students earned such grades by 16 percentage points.
And what drove that increase? First-generation students who got the chatbot reminders subsequently took the recommended actions — like completing assignments — at higher rates.
I was curious how the course’s instructor, Michael C. Evans, a senior lecturer in political science, viewed the experiment. Evans told me he’s always eager to work with the university’s student-success efforts. And the chatbots aren’t all that different from other forms of support built into the course: Evans sends batches of customized emails through the learning-management system already. Still, the reminder messages provided another layer of support, and in a different medium. The chatbots also let students pose questions, which are fielded by a combination of the system’s AI-enabled chatbot technology and a graduate teaching assistant. The teaching assistant will respond to questions that the AI does not yet recognize and also may monitor and follow up on AI responses to provide additional information.
Evans had several observations about giving students this kind of support in a course. First, he noted, the initial findings are from a fall semester — meaning many of the students in this intro course were in their first term of college. “We all remember what that was like,” he said. “A lot of your mind space is devoted to figuring out how to be a college student, but also, where do you go to eat? How do you find your classroom?” There’s a lot to keep track of. “Anybody in that state of mind is going to be prone to missing due dates; forgetting little logistical things.”
It stands to reason, he thinks, reminders would help.
Some professors view meeting deadlines as an important skill. If they do, Evans thinks, they should make that explicit to students, and support them in developing it.
And Evans disagrees with the interpretation that students’ ability to remember and meet deadlines demonstrates organizational aptitude. Instead, he thinks, “a lot of the students meeting the deadlines without assistance aren’t especially organized, either — they just have fewer concerns in their life” like family and work responsibilities.
It’s not about whether students are organized, in other words. It’s about how much they have to organize. If that’s the real difference between meeting and missing deadlines, then sending reminders is a step toward improving equity.
What do you think of nudging students in a classroom context? Have you tried anything along those lines, and if so, how did it work? Share your thoughts with me at beckie.supiano@chronicle.com and they may appear in a future newsletter.
I would definitely need some technical help in implementing chatbots in my courses. I do think it would be particularly helpful in my online courses, though.
Allison Swanson, the AT-IT Coordinator and Accessibility Facilitator in the Assistive Technology Center asked me to forward this announcement:
The Assistive Technology Resource Center (ATRC) is excited to launch the Electronic Accessibility Allies Team! Over the past few years, as we have advocated for accessibility at CSU, we’ve met people across our university who are invested in and excited about their part in creating a more inclusive campus. Gradually we realized that these people—our Accessibility Allies—needed a centralized forum to connect, brainstorm ideas, ask questions, and share success stories from their units. This Team provides an opportunity for collaboration that is modeled after similar communities and liaison groups at exemplary institutions of higher education.
We hope you’ll join us! We also encourage you to forward this invitation to anyone else you think might be interested.
If you have any questions, please reach out to our facilitators, Allison Swanson (allison.r.swanson@colostate.edu) and Chelsea Hansen (chelsea.hansen2@colostate.edu).
I hope you all finish out your spring semester strong and have a productive and regenerative summer. I sincerely hope we put the worst of this pandemic behind us while learning some lasting lessons.
Cheers, Paul
Paul Laybourn (he/him/his)
Professor, Biochemistry and Molecular Biology
Director, W2R S-STEM Program
Director, NoCo B2B Program
Director, REU Site in Molecular Biosciences
paul.laybourn@colostate.edu
970-491-5100
