Leveraging LLMs for student feedback in introductory data science courses
This page contains materials for the “Leveraging LLMs for student feedback in introductory data science courses” breakout session at USCOTS 2025.
Session abstract
Over the last few years, a considerable challenge for learners and teachers of data science courses has been the proliferation of the use of large language model (LLM) based tools (e.g., Chat GPT, CoPilot, etc.) in generating answers to assessment questions. In some cases, this might be considered a violation of an academic conduct policy, but this is not what this breakout session is about. Instead, in this session, we will share tools built for helping students leverage LLMs to get immediate feedback on their work in an effort to convince them to give it a try themselves instead of immediately turning to LLMs to generate answers for high-stakes assessments. This tool, an R package, allows students to develop their own answers and then use an LLM to evaluate their answers against a rubric that is not, at the outset, visible to them, and receive immediate, detailed feedback on their work. We will discuss details of decisions made when building the tool, both the backend and the user interface, challenges around evaluations that are not done correctly by the LLM, and student feedback from the first set of users. Finally, we will touch on how this tool can be incorporated into the assessment structure of a data science course as a low-stakes assessment tool and discuss ethical considerations for using a tool that leverages LLMs, especially as part of the formal assessment structure of a course.The slides are below and the hands-on activity can be accessed via the sidebar on the left.
Image in the website logo as well as in the slides is generated with ChatGPT with the prompt “Generate an image with a robot helping children. The kids should be looking at a laptop. And make the image rectangle, with height less than width.” https://chatgpt.com/share/682c8623-b100-8000-972c-e7384801436f