SYSTEMS AND METHODS FOR HIERARCHICAL MULTI-LABEL CONTRASTIVE LEARNING
Embodiments described herein provide a hierarchical multi-label framework to learn an embedding function that may capture the hierarchical relationship between classes at different levels in the hierarchy. Specifically, supervised contrastive learning framework may be extended to the hierarchical multi-label setting. Each data point has multiple dependent labels, and the relationship between labels is represented as a hierarchy of labels. The relationship between the different levels of labels may then be learnt by a contrastive learning framework.
1 . A method for hierarchical multi-label contrastive learning, the method comprising:
receiving a training dataset of image samples, wherein the training data set comprises at least one image sample that is associated with a set of hierarchical labels at a plurality of levels;
selecting, for the at least one image sample, a plurality of corresponding positive image samples corresponding to the plurality of levels in the set of hierarchical labels and a plurality of negative image samples;
generating, by a machine learning model, contrastive outputs in response to a plurality of positive input pairs formed by the at least one image sample and the plurality of corresponding positive image samples and a plurality of negative input pairs formed by the at least one image sample and the plurality of negative image samples;
computing a contrastive pair loss at a certain level based on a similarity between the contrastive outputs corresponding to the certain level;
computing a training objective by aggregating computed contrastive pair losses across the plurality of levels; and
updating the machine learning model based on the training objective.
2 . The method of claim 1 , wherein the set of hierarchical labels takes a form of a tree structure according to the plurality of levels, and wherein the tree structure has a root corresponding to a broadest label of the set of hierarchical labels.
3 . The method of claim 2 , further comprising:
randomly selecting an anchor image sample from the training dataset;
determining an anchor set of hierarchical labels in the tree structure associated with anchor image sample;
randomly selecting, for the anchor image sample at a first level from the plurality of levels, a first positive image sample that shares common label ancestry from the root up to the first level with the anchor image sample; and
forming a first positive pair from the anchor image sample and the first positive image sample.
4 . The method of claim 3 , further comprising:
randomly selecting, for the anchor image sample at another level from the plurality of levels, another positive image sample until positive image samples according to the plurality of levels have been sampled.
5 . The method of claim 4 , further comprising:
randomly selecting another anchor image until a batch of training image samples have been sampled in a training epoch.
6 . The method of claim 3 , further comprising:
generating, by an encoder, an anchor representation and a first positive representation from the anchor image sample and the first positive image sample, respectively; and
computing a first pair loss corresponding to the first positive pair based on a distance between the anchor representation and the first positive representation in a feature space.
7 . The method of claim 6 , further comprising:
computing a loss objective based at least in part on summing pair losses over positive image samples at each level and over the plurality of levels.
8 . The method of claim 6 , further comprising:
determining, at each level from the plurality of levels, a respective maximum pair loss among positive pairs at the respective level subject to a condition that the respective maximum pair loss is no less than another maximum pair loss corresponding to a lower label level; and
computing a loss objective based at least in part on summing maximum pair losses over positive image samples at each level and among the plurality of levels.
9 . A system for hierarchical multi-label contrastive learning, the system comprising:
a memory storing a plurality of processor-executable instructions for hierarchical multi- label contrastive learning; and
one or more hardware processors reading the plurality of processor-executable instructions to perform operations comprising:
receiving a training dataset of image samples, wherein the training data set comprises at least one image sample that is associated with a set of hierarchical labels at a plurality of levels;
selecting, for the at least one image sample, a plurality of corresponding positive image samples corresponding to the plurality of levels in the set of hierarchical labels and a plurality of negative image samples;
generating, by a machine learning model, contrastive outputs in response to a plurality of positive input pairs formed by the at least one image sample and the plurality of corresponding positive image samples and a plurality of negative input pairs formed by the at least one image sample and the plurality of negative image samples;
computing a contrastive pair loss at a certain level based on a similarity between the contrastive outputs corresponding to the certain level;
computing a training objective by aggregating computed contrastive pair losses across the plurality of levels; and
updating the machine learning model based on the training objective.
10 . The system of claim 9 , wherein the set of hierarchical labels takes a form of a tree structure according to the plurality of levels, and wherein the tree structure has a root corresponding to a broadest label of the set of hierarchical labels.
11 . The system of claim 10 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
randomly selecting an anchor image sample from the training dataset;
determining an anchor set of hierarchical labels in the tree structure associated with anchor image sample;
randomly selecting, for the anchor image sample at a first level from the plurality of levels, a first positive image sample that shares common label ancestry from the root up to the first level with the anchor image sample; and
forming a first positive pair from the anchor image sample and the first positive image sample.
12 . The system of claim 11 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
randomly selecting, for the anchor image sample at another level from the plurality of levels, another positive image sample until positive image samples according to the plurality of levels have been sampled.
13 . The system of claim 12 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
randomly selecting another anchor image until a batch of training image samples have been sampled in a training epoch.
14 . The system of claim 11 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
generating, by an encoder, an anchor representation and a first positive representation from the anchor image sample and the first positive image sample, respectively; and
computing a first pair loss corresponding to the first positive pair based on a distance between the anchor representation and the first positive representation in a feature space.
15 . The system of claim 14 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
computing a loss objective based at least in part on summing pair losses over positive image samples at each level and over the plurality of levels.
16 . The system of claim 14 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
determining, at each level from the plurality of levels, a respective maximum pair loss among positive pairs at the respective level subject to a condition that the respective maximum pair loss is no less than another maximum pair loss corresponding to a lower label level; and
computing a loss objective based at least in part on summing maximum pair losses over positive image samples at each level and among the plurality of levels.
17 . A processor-readable non-transitory storage medium storing a plurality of processor- executable instructions for hierarchical multi-label contrastive learning, the plurality of processor-executable instructions being executed by one or more processors to perform operations comprising:
receiving a training dataset of image samples, wherein the training data set comprises at least one image sample that is associated with a set of hierarchical labels at a plurality of levels;
selecting, for the at least one image sample, a plurality of corresponding positive image samples corresponding to the plurality of levels in the set of hierarchical labels and a plurality of negative image samples;
generating, by a machine learning model, contrastive outputs in response to a plurality of positive input pairs formed by the at least one image sample and the plurality of corresponding positive image samples and a plurality of negative input pairs formed by the at least one image sample and the plurality of negative image samples;
computing a contrastive pair loss at a certain level based on a similarity between the contrastive outputs corresponding to the certain level;
computing a training objective by aggregating computed contrastive pair losses across the plurality of levels; and
updating the machine learning model based on the training objective.
18 . The processor-readable non-transitory storage medium of claim 17 , wherein the operations comprise:
randomly selecting an anchor image sample from the training dataset;
determining an anchor set of hierarchical labels in the tree structure associated with anchor image sample;
randomly selecting, for the anchor image sample at a first level from the plurality of levels, a first positive image sample that shares common label ancestry from the root up to the first level with the anchor image sample;
forming a first positive pair from the anchor image sample and the first positive image sample;
randomly selecting, for the anchor image sample at another level from the plurality of levels, another positive image sample until positive image samples according to the plurality of levels have been sampled; and
randomly selecting another anchor image until a batch of training image samples have been sampled in a training epoch.
19 . The processor-readable non-transitory storage medium of claim 17 , wherein the operations further comprise:
generating, by an encoder, an anchor representation and a first positive representation from the anchor image sample and the first positive image sample, respectively; and
computing a first pair loss corresponding to the first positive pair based on a distance between the anchor representation and the first positive representation in a feature space.
20 . The processor-readable non-transitory storage medium of claim 19 , wherein the operations further comprise:
determining, at each level from the plurality of levels, a respective maximum pair loss among positive pairs at the respective level subject to a condition that the respective maximum pair loss is no less than another maximum pair loss corresponding to a lower label level; and
computing a loss objective based at least in part on summing maximum pair losses over positive image samples at each level and among the plurality of levels.