Gender Inference: Can ChatGPT Outperform Common Commercial Tools?

Abstract

An increasing number of studies use gender information to understand phenomena such as gender bias, inequity in access and participation, and the impact of the COVID-19 pandemic response. Unfortunately, most datasets do not include self-reported gender information, which makes it necessary for researchers to infer gender from other information, such as from names or names and country information. In this paper, we compare the performance of the new generative Artificial Intelligence (AI) tool ChatGPT with three traditional commercially available list-based and machine learning-based gender inference tools—Namsor, Gender-API, and genderize.io—on a unique dataset. Specifically, we use a large Olympic athlete dataset and report how variations in the input (e.g., first name and first & last name, with and without country information) impact the accuracy of their predictions. We find that Namsor is the best traditional commercially available tool. However, ChatGPT performs at least as well as Namsor and often outperforms it, especially for the female sample when country and/or last name information is available. We conclude ChatGPT may be a cost-effective tool for gender prediction.

Publication
CASCON 2023–IBM TechXchange, pp. 1–6