Systems and methods for determining and using a multidimensional measure of apparent skin color
Systems and methods can compute a multidimensional score of apparent skin color to capture skin color variation and provide a comprehensive representation of its constitutive complexity. The multidimensional apparent skin color score can be used to reveal biases related to skin color in image datasets and computer vision models. The multidimensional skin color score can focus on the perceptual lightless, L*, as a measure of skin tone ranging from light to dark, and the hue angle h*, as a measure of skin hue ranging from red to yellow. The multidimensional skin color scale offers a more representative assessment to surface socially relevant biases due to skin color effects in computer vision. This could help to (i) enhance the diversity in the data collection process, by encouraging specifications to better represent skin color variability; and (ii) improve the identification of dataset and model biases in fairness benchmarking, by highlighting their limitations.
1 . A computer-implemented method for determining an apparent skin color of a face image, comprising:
receiving an input image;
segmenting skin pixels in the input image;
extracting a perceptual lightness, L*, value, a green-red opponent color value, a*, and a blue-yellow opponent color value in each segmented skin pixel;
applying a clustering algorithm to cluster the segmented skin pixels;
computing a histogram of a distribution obtained from the clustering algorithm; and
assigning a scalar value for L* and for a hue angle, h*, based on a mode of the histogram, for each cluster.
2 . The computer-implemented method of claim 1 , further comprising determining a final scalar score for the input image by averaging the scalar values of each cluster, normalized by a pixel size of each cluster.
3 . The computer-implemented method of claim 1 , further comprising excluding clusters that have an L* value below a predetermined minimum.
4 . The computer-implemented method of claim 1 , wherein:
the clustering algorithm provides five clusters; and
out of the five clusters, three clusters with the greatest L* value are retained to compute the L* and h* scalar values.
5 . The computer-implemented method of claim 1 , wherein a number of bins of the histogram is determined by a Sturges formula.
6 . The computer-implemented method of claim 1 , wherein the segmentation of the skin pixels is performed with a skin segmentation machine learning model.
7 . A computer-implemented method for determining bias in a face image dataset, comprising:
determining an apparent skin color of each face image in the face image dataset by:
receiving an input image having the face image;
segmenting skin pixels in the input image;
extracting a perceptual lightness, L*, value, a green-red opponent color value, a*, and a blue-yellow opponent color value in each segmented skin pixel;
applying a clustering algorithm to cluster the segmented skin pixels;
computing a histogram of a distribution obtained from the clustering algorithm;
assigning a scalar value for L* and for a hue angle, h*, based on a mode of the histogram, for each cluster; and
determining a final scalar score for the input image by averaging the scalar values of each cluster, normalized by a pixel size of each cluster;
determining the bias in the face image dataset with respect to light and dark skin tones by calculating a number of face images having the L* scalar value greater than and less than a first predetermined threshold; and
determining the bias in the face image dataset with respect to skin hue by calculating a number of face images having the h* scalar value greater than and less than a second predetermined threshold.
8 . The computer-implemented method of claim 7 , further comprising excluding clusters that have an L* value below a predetermined minimum.
9 . The computer-implemented method of claim 7 , wherein:
the clustering algorithm provides five clusters; and
out of the five clusters, three clusters with the greatest L* value are retained to compute the L* and h* scalar values.
10 . The computer-implemented method of claim 7 , wherein a number of bins of the histogram is determined by a Sturges formula.
11 . The computer-implemented method of claim 7 , wherein the segmentation of the skin pixels is performed with a skin segmentation machine learning model.
12 . The computer-implemented method of claim 7 , further comprising:
generating images with a generative adversarial network and a diffusion model; and
determining the apparent skin color of each face image in the generated images; and
determining whether the determined bias is present in the generated images.
13 . A computer-implemented method for determining bias in a computer vision-based model, comprising:
determining an apparent skin color of each face image in the face image dataset by:
receiving an input image having the face image;
segmenting skin pixels in the input image;
extracting a perceptual lightness, L*, value, a green-red opponent color value, a*, and a blue-yellow opponent color value in each segmented skin pixel;
applying a clustering algorithm to cluster the segmented skin pixels;
computing a histogram of a distribution obtained from the clustering algorithm;
assigning a scalar value for L* and for a hue angle, h*, based on a mode of the histogram, for each cluster; and
determining a final scalar score for the input image by averaging the scalar values of each cluster, normalized by a pixel size of each cluster;
using the face image dataset in a machine learning model; and
determining whether an output of the machine learning model includes a bias based on at least one of the perceptual lightness value or the hue angle value.
14 . The computer-implemented method of claim 13 , wherein the machine learning model is a face cropping model that selects one face image from a group of two face images, the method further comprising:
measuring a probability of the one face image to be selected by the face cropping model;
comparing the perceptual lightness value and the hue angle value of the two face images; and
determining a bias in at least one of the perceptual lightness value or the hue angle value when the face cropping model selects the one face image.
15 . The computer-implemented method of claim 13 , wherein the machine learning model is a face recognition model that determines whether two face images are from the same person, the method further comprising:
determining a percentage of correct determinations that the face recognition model correctly determine whether the two face images are from the same person; and
determining bias between the percentage of correct determines and at least one of the perceptual lightness value or the hue angle value.
16 . The computer-implemented method of claim 13 , wherein the machine learning model is an attribute prediction model that determines whether the face image includes a selected attribute, the method further comprising:
manipulating the face image dataset by changing a skin tone and a skin hue of each image therein; and
observing a causal effect of the manipulating on a performance of correct prediction of the selected attribute with the attribute prediction model.
17 . The computer-implemented method of claim 13 , further comprising excluding clusters that have an L* value below a predetermined minimum.
18 . The computer-implemented method of claim 13 , wherein:
the clustering algorithm provides five clusters; and
out of the five clusters, three clusters with the greatest L* value are retained to compute the L* and h* scalar values.