Methods and apparatus for visual-aware hierarchy-based object recognition

1 . A computerized method for training a machine learning model for recognizing objects based on semantic and visual information associated with the objects, the method comprising:

accessing a set of digital images, each digital image comprising an object associated with a semantic category of the object, wherein the semantic category is one of a plurality of semantic categories associated with the set of digital images, and wherein each digital image is labeled with data indicative of the semantic category to which the digital image is associated;

determining, using a trained machine learning model, a visual feature vector for each digital image of the set of digital images, wherein the visual feature vector comprises a value for each of a plurality of visual features;

for each semantic category of the plurality of semantic categories, calculating, for each visual feature of the plurality of visual features, a separate probability distribution of the values of the visual feature in the visual feature vectors of each image in the set of digital images associated with the semantic category;

determining, based on the probability distributions, data indicative of similarities between digital images associated with different semantic categories of the plurality of semantic categories;

grouping, based on the data indicative of similarities between the digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories;

classifying, using the machine learning model, each digital image of the set of digital images, said classifying comprising:

determining, using a first output layer of the machine learning model, a predicted semantic category of the plurality of semantic categories for said digital image, and

determining, using a second output layer of the machine learning model, a predicted cluster for said digital image; and

training the machine learning model using a loss function, wherein the loss function incorporates data indicative of (i) whether the predicted semantic category for each digital image corresponds to the semantic category associated with said digital image and (ii) whether the predicted cluster for said digital image corresponds to the cluster into which the semantic category associated with said digital image was grouped.

2 . The method of claim 1 , wherein, for each image of the set of digital images, the object is food and the semantic category is a food category of the food.

3 . The method of claim 1 , wherein the visual feature vector comprises a 1024-dimension space vector, wherein each dimension of the visual feature vector corresponds to a different visual feature.

4 . The method of claim 1 , wherein the calculated probability distributions are Gaussian distributions.

5 . The method of claim 1 , wherein determining data indicative of similarities between digital images of different semantic categories comprises:

generating a similarity matrix comprising a number of rows and a number of columns that are both equal to a total number of the plurality of semantic categories, such that each row and each column are associated with a semantic category of the plurality of semantic categories; and

for each matrix entry in the similarity matrix, determining data indicative of a similarity between the visual feature vectors of (a) a first set of digital images of the set of digital images associated with the semantic category of the row of the matrix entry and (b) a second set of digital images of the set of digital images associated with the semantic category of the column of the matrix entry.

6 . The method of claim 5 , wherein:

the calculated probability distributions are Gaussian distributions; and

determining, for each matrix entry in the similarity matrix, the data indicative of the similarity between the visual feature vectors comprises:

for each visual feature of the plurality of visual features, generating an overlap coefficient indicative of an overlap between the Gaussian distributions for the visual feature of (a) the first set of digital images and (b) the second set of digital images,

generating an averaged overlap coefficient by averaging the overlap coefficients, and

providing the averaged overlap coefficient as the data indicative of the similarity between the visual feature vectors.

7 . The method of claim 6 , wherein grouping, based on the data indicative of similarities between digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories comprises:

grouping, using affinity propagation, the plurality of semantic categories into clusters of semantic categories based on the averaged overlap coefficients of the similarity matrix.

8 . A non-transitory computer-readable media comprising instructions for training a machine learning model for recognizing objects based on semantic and visual information associated with the objects, wherein the instructions, when executed by one or more processors on a computing device, are operable to cause the one or more processors to:

access a set of digital images, each digital image comprising an object associated with a semantic category of the object, wherein the semantic category is one of a plurality of semantic categories associated with the set of digital images, and wherein each digital image is labeled with data indicative of the semantic category to which the digital image is associated;

determine, using a trained machine learning model, a visual feature vector for each digital image of the set of digital images, wherein the visual feature vector comprises a value for each of a plurality of visual features;

for each semantic category of the plurality of semantic categories, calculate, for each visual feature of the plurality of visual features, a separate probability distribution of the values of the visual feature in the visual feature vectors of each image in the set of digital images associated with the semantic category;

determine, based on the probability distributions, data indicative of similarities between digital images associated with different semantic categories of the plurality of semantic categories;

group, based on the data indicative of similarities between the digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories;

classify, using the machine learning model, each digital image of the set of digital images, said classifying comprising:

determining, using a first output layer of the machine learning model, a predicted semantic category of the plurality of semantic categories for said digital image, and

determining, using a second output layer of the machine learning model, a predicted cluster for said digital image; and

training the machine learning model using a loss function, wherein the loss function incorporates data indicative of (i) whether the predicted semantic category for each digital image corresponds to the semantic category associated with said digital image and (ii) whether the predicted cluster for said digital image corresponds to the cluster into which the semantic category associated with said digital image was grouped.

9 . The non-transitory computer-readable media of claim 8 , wherein, for each image of the set of digital images, the object is food and the semantic category is a food category of the food.

10 . The non-transitory computer-readable media of claim 8 , wherein the visual feature vector comprises a 1024-dimension space vector, wherein each dimension of the visual feature vector corresponds to a different visual feature.

11 . The non-transitory computer-readable media of claim 8 , wherein the calculated probability distributions are Gaussian distributions.

12 . The non-transitory computer-readable media of claim 8 , wherein determining data indicative of similarities between digital images of different semantic categories comprises:

generating a similarity matrix comprising a number of rows and a number of columns that are both equal to a total number of the plurality of semantic categories, such that each row and each column are associated with a semantic category of the plurality of semantic categories; and

for each matrix entry in the similarity matrix, determining data indicative of a similarity between the visual feature vectors of (a) a first set of digital images of the set of digital images associated with the semantic category of the row of the matrix entry and (b) a second set of digital images of the set of digital images associated with the semantic category of the column of the matrix entry.

13 . The non-transitory computer-readable media of claim 12 , wherein:

the calculated probability distributions are Gaussian distributions; and

determining, for each matrix entry in the similarity matrix, the data indicative of the similarity between the visual feature vectors comprises:

for each visual feature of the plurality of visual features, generating an overlap coefficient indicative of an overlap between the Gaussian distributions for the visual feature of (a) the first set of digital images and (b) the second set of digital images,

generating an averaged overlap coefficient by averaging the overlap coefficients, and

providing the averaged overlap coefficient as the data indicative of the similarity between the visual feature vectors.

14 . The non-transitory computer-readable media of claim 13 , wherein grouping, based on the data indicative of similarities between digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories comprises:

grouping, using affinity propagation, the plurality of semantic categories into clusters of semantic categories based on the averaged overlap coefficients of the similarity matrix.

15 . A system for training a machine learning model for recognizing objects based on semantic and visual information associated with the objects, the system comprising:

a memory storing instructions; and

a processor configured to execute the instructions to:

access a set of digital images, each digital image comprising an object associated with a semantic category of the object, wherein the semantic category is one of a plurality of semantic categories associated with the set of digital images, and wherein each digital image is labeled with data indicative of the semantic category to which the digital image is associated;

determine, using a trained machine learning model, a visual feature vector for each digital image of the set of digital images, wherein the visual feature vector comprises a value for each of a plurality of visual features;

for each semantic category of the plurality of semantic categories, calculate, for each visual feature of the plurality of visual features, a separate probability distribution of the values of the visual feature in the visual feature vectors of each image in the set of digital images associated with the semantic category;

determine, based on the probability distributions, data indicative of similarities between digital images associated with different semantic categories of the plurality of semantic categories;

group, based on the data indicative of similarities between the digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories;

classify, using the machine learning model, each digital image of the set of digital images, said classifying comprising:

determining, using a first output layer of the machine learning model, a predicted semantic category of the plurality of semantic categories for said digital image, and

determining, using a second output layer of the machine learning model, a predicted cluster for said digital image; and

training the machine learning model using a loss function, wherein the loss function incorporates data indicative of (i) whether the predicted semantic category for each digital image corresponds to the semantic category associated with said digital image and (ii) whether the predicted cluster for said digital image corresponds to the cluster into which the semantic category associated with said digital image was grouped.

16 . The system of claim 15 , wherein, for each image of the set of digital images, the object is food and the semantic category is a food category of the food.

17 . The system of claim 15 , wherein the visual feature vector comprises a 1024-dimension space vector, wherein each dimension of the visual feature vector corresponds to a different visual feature.

18 . The system of claim 15 , wherein the calculated probability distributions are Gaussian distributions.

19 . The system of claim 15 , wherein determining data indicative of similarities between digital images of different semantic categories comprises:

generating a similarity matrix comprising a number of rows and a number of columns that are both equal to a total number of the plurality of semantic categories, such that each row and each column are associated with a semantic category of the plurality of semantic categories; and

for each matrix entry in the similarity matrix, determining data indicative of a similarity between the visual feature vectors of (a) a first set of digital images of the set of digital images associated with the semantic category of the row of the matrix entry and (b) a second set of digital images of the set of digital images associated with the semantic category of the column of the matrix entry.

20 . The system of claim 19 , wherein:

the calculated probability distributions are Gaussian distributions; and

determining, for each matrix entry in the similarity matrix, the data indicative of the similarity between the visual feature vectors comprises:

for each visual feature of the plurality of visual features, generating an overlap coefficient indicative of an overlap between the Gaussian distributions for the visual feature of (a) the first set of digital images and (b) the second set of digital images,

generating an averaged overlap coefficient by averaging the overlap coefficients, and

providing the averaged overlap coefficient as the data indicative of the similarity between the visual feature vectors.

21 . The system of claim 20 , wherein grouping, based on the data indicative of similarities between digital images associated with different semantic categories, the plurality of semantic categories into clusters of semantic categories comprises:

grouping, using affinity propagation, the plurality of semantic categories into clusters of semantic categories based on the averaged overlap coefficients of the similarity matrix.