Sub-object segmentation
A method includes receiving an object mask of an object in an image. The method further includes generating a mask of a sub-object in the image using a machine learning model configured to receive the mask of the object. A first branch of the machine learning model predicts whether a pixel of the image belongs to a sub-object.
1 . A method comprising:
receiving an object mask of an object in an image; and
generating a mask of a sub-object in the image using a machine learning model configured to receive the object mask, wherein a first branch of the machine learning model predicts a likelihood of a pixel of the image belonging to a sub-object mask, wherein a second branch of the machine learning model predicts a likelihood of the pixel of the image belonging to the sub-object.
2 . The method of claim 1 , further comprising:
executing a clustering technique to cluster a plurality of features of the image into one or more clusters, wherein the one or more clusters correspond to one or more sub-objects of the object.
3 . The method of claim 2 , wherein the machine learning model is trained using pseudo labels associated with each of the one or more clusters, the pseudo labels being used as a class agnostic label.
4 . The method of claim 1 , further comprising:
overlaying a visual indicator over the sub-object, the overlayed visual indicator corresponding to the mask of the sub-object; and
displaying the overlayed visual indicator over the sub-object.
5 . The method of claim 1 , further comprising:
displaying a plurality of sub-objects of the object; and
receiving a selection of a sub-object from the plurality of sub-objects.
6 . The method of claim 1 , further comprising:
detecting an event associated with the sub-object in the image; and
triggering, in response to the event, the machine learning model to generate the mask of the sub-object.
7 . A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
receiving an object mask of an object in an image; and
generating a mask of a sub-object in the image using a machine learning model configured to receive the object mask, wherein a first branch of the machine learning model predicts a likelihood of a pixel of the image belonging to the sub-object, wherein a second branch of the machine learning model predicts a likelihood of the pixel of the image belonging to a sub-object mask.
8 . The system of claim 7 , wherein the processing device performs further operations comprising:
executing a clustering technique to cluster a plurality of features of the image into one or more clusters, wherein the one or more clusters correspond to one or more sub-objects of the object.
9 . The system of claim 8 , wherein the machine learning model is trained using pseudo labels associated with each of the one or more clusters, the pseudo labels being used as a class agnostic label.
10 . The system of claim 7 , wherein the processing device performs further operations comprising:
overlaying a visual indicator over the sub-object, the overlayed visual indicator corresponding to the mask of the sub-object; and
displaying the overlayed visual indicator over the sub-object.
11 . The system of claim 7 , wherein the processing device performs further operations comprising:
displaying a plurality of sub-objects of the object; and
receiving a selection of a sub-object from the plurality of sub-objects.
12 . The system of claim 7 , wherein the processing device performs further operations comprising:
detecting an event associated with the sub-object in the image; and
triggering, in response to the event, the machine learning model to generate the mask of the sub-object.
13 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
receiving an object mask of an object in an image;
predicting, by a first branch of a machine learning model, a likelihood of a pixel of the image belonging to a sub-object mask;
predicting, by a second branch of the machine learning model, a likelihood of the pixel of the image belonging to a sub-object using an objectness score; and
determining, using the likelihood of the pixel of the image belonging to the sub-object mask and the likelihood of the pixel of the image belonging to the sub-object, a mask of the sub-object.
14 . The non-transitory computer-readable medium of claim 13 , storing instructions that further cause the processing device to perform operations comprising:
executing a clustering technique in a feature space to cluster a plurality of features of the image into one or more clusters, wherein the one or more clusters correspond to one or more sub-objects of the object.
15 . The non-transitory computer-readable medium of claim 14 , wherein the feature space is an image feature space determined by a pixel-level module, a per segment embedding space determined by a transformer module, a per pixel embedding space determined by the pixel-level module, or a mask embedding space, determined by a multi layer perceptron.
16 . The non-transitory computer-readable medium of claim 14 , wherein the machine learning model is trained using pseudo labels associated with each of the one or more clusters, the pseudo labels being used as a class agnostic label.
17 . The non-transitory computer-readable medium of claim 16 , wherein training further includes:
adjusting an error of the machine learning model by a first amount responsive to a pseudo label being received as a training input; and
adjusting the error of the machine learning model by a second amount responsive to a ground truth label being received as the training input.
18 . The non-transitory computer-readable medium of claim 13 , wherein:
overlaying a visual indicator over the sub-object, the overlayed visual indicator corresponding to the mask of the sub-object; and
displaying the overlayed visual indicator over the sub-object.