Abstract

The rapid rise of generative language models (LMs) has raised concerns about social biases and their impact on diverse users. While prior studies focus on explicit identity prompts, research shows that discrimination can also occur without them. We extend this work by analyzing LM behavior under open-ended prompting, a more natural use case. Our study of five leading LMs reveals that minoritized individuals—across race, gender, and sexual orientation—face significant harms through omission, subordination, and stereotyping in 500,000 generated texts. These identities are far more likely to be portrayed in subordinated ways than in representative or empowering ones. We also observe stereotypes known to cause psychological harm. These findings underscore the urgent need for critical and inclusive AI literacy to mitigate discriminatory impacts on students and educators.