How students’ GenAI skills and reflection affect assignment instructions

November 2022: ChatGPT rapidly emerges as the next big disruptor in higher education. On campuses across the US, the primary feelings are scepticism and fear of cheating, but pushing past that is the notion that this technology could be harnessed to benefit education.

Spring 2023: At Georgia Institute of Technology, our conversations and workshops on generative AI (GenAI) focus on how faculty can use it in course design, assignment creation, personalised learning efforts and more. Fear and scepticism still exist but don’t obstruct brainstorming efforts. In the summer, we see instructors’ responses range from dipping toes into the AI water and using it to create rubrics, case studies and other standard course content to diving in headfirst and using GenAI to produce entire courses.

Fall 2023: Many employers of future graduates want students to gain knowledge and experience using GenAI tools while in their degree programmes. Thinking shifts from students wanting to use GenAI to cheat to students needing to learn about GenAI to succeed. The professors at our institution are beginning to embrace the idea that they should support the correct usage of GenAI in their classrooms. 

• AI can help fix student evaluations
• Spotlight collection: AI and assessment in higher education
• How can we teach AI literacy skills?
• Resource collection: How to build data literacy on campus

Here lies the challenge: how much direction should you include in GenAI-inclusive assignments? Previously, instructors had to balance assignment guidelines with student creativity, so students could create a unique submission while remaining within the assignment objectives. Now the additional task is finding the right amount of guidance to ensure students can effectively use GenAI beyond simply copying and pasting predefined prompts.

Creating GenAI assignments

How to create assignments using GenAI is one of the questions that co-authors Ilya Gokhman and Vincent Spezzo have addressed. Students in Gokhman’s public policy course worked in groups of four to complete project-based tasks and provide peer feedback to their team members at four points during the semester. The idea was to have students use GenAI as a leadership and collaboration coach to help them process and reflect on peer feedback. GenAI was used in three of the four feedback phases (students completed the first reflection unassisted). For the remaining phases, students were: 1) instructed to use GenAI with no further guidance, 2) given detailed instructions on how to use GenAI, including suggested prompts, and 3) instructed to use GenAI how they wanted, whether that was to use the instructors suggested prompts or their own. 

Students divided on using GenAI

Students were surveyed on a six-point Likert scale to determine their experience using GenAI in their assignments and how it impacted their learning (see list below). This included a self-rating on their prior experience using GenAI that included options of “a lot”, “some”, “little” and “none”.  From the 72 participants, several novel insights were gleaned. The most significant finding was a clear division in students’ experiences using GenAI for assignments between the two groups at opposite ends of the prior-usage spectrum. Those students who had the most prior experience rated several items significantly higher than those who came into the class never having used GenAI before. This was true despite very detailed instructions and prompt examples being added to the third and fourth assignments. 

• I would rather use GenAI than complete feedback review with another person: A lot M=4.7, None M=2.78 
• I felt using GenAI helped me in learning the course material: A lot M=4.8, None M=3.06 
• I felt using GenAI increased my motivation to complete assignments: A lot M=4.8, None M=2.83
• Overall, I felt using GenAI had a positive impact on my course experience: A lot M=5.4, None M=3.89

Two things worth noting are: 1) while students with more experience rated items significantly higher, students with no prior experience generally still rated items around a three on the six-point scale, and 2) students who fell into the two middle groupings were not shown to be significantly different from the two extremes on almost all questions. 

Students also responded to open-ended questions, with 30 per cent stating that the assignments could be improved by more frequent GenAI usage, 25 per cent commented that GenAI was useful in generating ideas and expanding their perspectives, and 20 per cent indicated a desire for more detailed instructions on using GenAI for the assignment. 

Addressing students’ differences in experience using GenAI 

Results pointed to a difference in instructional needs between students with no GenAI experience and those with a lot of experience. One could mistakenly assume the number of students with little or no experience will decrease as use of these technologies becomes more widespread. However, the ability to use GenAI is likely more akin to time management, studying and a host of other learning skills that students need support and practice with before mastering. Coupling this with the current lack of adoption at K-12, it is very likely that inequity with prior usage of GenAI will exist for some time.

Two actionable practices that can address this inequality:

1. Include more detailed instruction and prompt examples for assignments. While not all students need this, at least 20 per cent of the students surveyed indicated they wanted even more direction than was provided, and there was no indication that the additional instructions negatively impacted students with prior experience. Part of an equitable framework is to ensure that those who need the additional support have it available, so including optional additional guidelines may be the way to go with GenAI assignments for now.
2. Create and include a lesson or optional module for teaching students how to use GenAI effectively within your course or discipline. From this study, it seems that simply including more examples and instructions was not enough for some students. To address the gap of experience, students seem to need support and exposure to the basics of using GenAI that goes beyond creating good prompts. There is already discussion of including such experiences in freshmen seminar courses, but until then it will be up to instructors to help bridge the gap for students who have yet to learn how to use GenAI in ways that will benefit their education. 

By using these practices, the intent is that students coming into a course with little or no prior GenAI experience can be brought up to speed and benefit at near the same level as those who have had a lot of experience using the tools. Conducting a start-of-semester survey is a good way to identify students who need additional resources and ensure they are directed to access them. While this means yet another task for instructors, the benefits to student learning and the expectations of future employers make this worth taking on.

Vincent Spezzo is assistant director of teaching and learning online in the Center for Teaching and Learning, and Ilya Gokhman is faculty lead for grand challenges in the Office of Leadership Education and Development, both at Georgia Institute of Technology.

If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.