Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts
Published in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024
Evaluating social biases and stereotypes in Natural Language Processing has gained significant momentum, but most resources originate from a single culture (primarily US-centric). This collaborative work analyzes practical hurdles when adapting and evaluating biases across multiple languages, demonstrating how cultural context fundamentally reshapes stereotypical associations and evaluation methodology.