Color-theoretic experiments to understand unequal gender classification accuracy from face images

Recent work shows unequal performance of commercial face classification services in the gender classification task across intersectional groups defined by skin type and gender. Accuracy on dark-skinned females is significantly worse than on any other group. We provide initial evidence that skin type alone is not the driver for this disparity by conducting novel stability experiments that vary an image's skin type via color-theoretic methods, namely luminance mode-shift and optimal transport. We evaluate the effect of skin type change on the gender classification decision of a pair of state-of-the-art commercial and open-source gender classifiers. The results raise the possibility that broader differences in ethnicity, as opposed to the skin type alone, are what contribute to unequal gender classification accuracy in face images.