Teaching Algorithm Auditing to Help Students Detect Bias in AI Outputs

As students integrate AI into their learning at an increasingly rapid pace, they need the tools to critically assess the outputs it provides—and how much they can trust them. That’s where algorithm auditing comes in.

“The question is, since we’re all interacting with AI systems, how do we know whether the output these systems generate is fair or biased?” says Yasmin Kafai, the Lori and Michael Milken President’s Distinguished Professor at Penn GSE. “A one-time interaction might raise some red flags, but in order to figure out whether there is a bias in the results, we need a much more systematic investigation.”

A leading learning designer and researcher, Kafai worked with Danaé Metaxa, the Raj and Neera Singh Term Assistant Professor in Penn Engineering’s Computer and Information Science department, and high school teachers to develop the “AI Auditing for High School” toolkit to help teachers introduce students to algorithmic bias and guide them through hands-on audits of real-world AI applications. Until very recently, she explains, most people thought that AI systems were too complex for the average person to understand how they work, and thus potential algorithmic bias was a fact of life.

“But algorithm audits have actually shown that you don’t need access to the algorithms, you don’t need access to the data used to train the system,” she says. “By just systematically investigating or auditing different inputs, you can see what kind of output the system generates, and then with some simple statistics you can make a judgment call: is this biased or not?”

Kafai provides the following advice for teachers looking to implement algorithm audits in their curriculum, whether in a computer science classroom or any subject dealing with media and information literacy.

It’s okay to keep datasets small

Students don’t typically work with as large of datasets as experts do. But Kafai and her colleagues found that when experts have conducted the same data analysis at scale, they have arrived at the same conclusions students did. So, you don’t need huge datasets to observe variations and differences in outputs.

Incorporate students’ perspectives and interests to diversify data

In Kafai’s research, students actually included variables that experts don’t usually consider, like including age in a gender bias analysis of AI image generator outputs for various professions. They also looked at professions that wouldn’t typically be found in an expert’s dataset, like tattoo artists and nail technicians. “They brought perspectives that have kind of been missing from that research,” Kafai notes. This diversity in students’ perspectives helped enrich the study—and it’s exactly what’s needed to evaluate AI systems.

“Many youths we worked with as advisors … said they felt validated in their expertise for the first time because these were topics and issues they were concerned with,” Kafai adds.

Be thoughtful about assignment design

Assignments should be tailored to your students’ level. While high school students can work with their own datasets, it may be better to create a light version for middle school students by implementing some data control. For example, teachers may want to create a constrained dataset and add biases into it for students to find, Kafai says. That way teachers will already know what the outcomes of students’ work should be.

Keep in mind that algorithmic systems change frequently, which means outputs of live data like online materials will likely be different from year to year. From a teaching perspective, developing materials is a big investment, and teachers may want datasets they can rely on for multiple years.

Keep the curriculum grounded in the scientific method

Many teachers and students are already familiar with the scientific method, so building off that provides students the scaffolding necessary to guide them through each activity in the curriculum.

Kafai also emphasizes the importance of the scientific method’s sixth step: reflecting on what was learned. Teachers play a critical role in this by organizing classroom discussions and getting students thinking about the process, outcomes, and possible improvements.

“I think this is a very empowering approach,” Kafai says. “You don’t need any programming skill. You don’t actually need technical skills. You just need to be systematic and rigorous, and that’s what we want students to be anyway.”

***

The curriculum resources kit Kafai and colleagues developed and co-designed with high school teachers can be accessed at https://bit.ly/k12aiaudit-lessons. Kafai’s collaborator, Danaé Metaxa, has also just published a book with colleagues on algorithm auditing titled Auditing AI (MIT Press, 2026).