Online adaptation of segmentation machine learning systems
Techniques and systems are provided for performing online adaptation of machine learning model(s). For example, a process may include obtaining features extracted from a image by a machine learning model during inference and determining, by the machine learning model based on the features during inference, a plurality of keypoint estimates in the image and/or a bounding region estimate associated with an object in the image. The process may further include generating pseudo-label(s) based on the plurality of keypoint estimates and/or the bounding region estimate. The process may include determining at least one self-supervised loss based on the plurality of keypoint estimates and/or the bounding region estimate. The process may further include adapting, based on the at least one self-supervised loss, parameter(s) of the machine learning model. The process may include generating, using the machine learning model with the adapted parameter(s), a segmentation mask for the image (or another image).
1 . A processor-implemented method, comprising:
obtaining features extracted from a first image by a machine learning model during inference;
determining, by the machine learning model based on the features during inference, at least one of a plurality of keypoint estimates in the first image or a bounding region estimate associated with an object in the first image;
generating one or more pseudo-labels based on at least one of the plurality of keypoint estimates or the bounding region estimate;
determining at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the bounding region estimate;
adapting, based on the at least one self-supervised loss, one or more parameters of the machine learning model; and
generating, using the machine learning model with the adapted one or more parameters, a segmentation mask for the first image or a second image;
wherein the features are extracted from the first image by a feature extraction engine of the machine learning model; and
wherein adapting, based on the at least one self-supervised loss, the one or more parameters of the machine learning model comprises adapting one or more parameters of the feature extraction engine.
2 . The method of claim 1 , further comprising:
processing, using the machine learning model with the adapted one or more parameters, the first image or a second image to generate updated features representing the first image or to generate features representing the second image.
3 . The method of claim 2 , wherein the segmentation mask is generated based on processing the updated features representing the first image or the features representing the second image.
4 . The method of claim 1 , wherein the plurality of keypoint estimates is determined in the first image by a keypoint detection engine of the machine learning model.
5 . The method of claim 1 , wherein the bounding region estimate is determined in the first image by an object detection engine of the machine learning model.
6 . The method of claim 1 , wherein the segmentation mask is generated by a segmentation mask engine of the machine learning model.
7 . The method of claim 1 , wherein the plurality of keypoint estimates includes at least one heat map indicating probabilities that portions of the first image correspond to a particular keypoint.
8 . The method of claim 7 , wherein the at least one heat map includes a plurality of heat maps, each heat map of the plurality of heat maps being associated with a respective keypoint of a plurality of keypoints.
9 . The method of claim 1 , wherein the plurality of keypoint estimates include coordinates associated with a plurality of keypoints.
10 . The method of claim 1 , wherein generating the one or more pseudo-labels comprises:
determining one or more keypoint estimates from the plurality of keypoint estimates that are greater than an individual keypoint score threshold; and
selecting the one or more keypoint estimates as the one or more pseudo-labels based on the one or more keypoint estimates being greater than the individual keypoint score threshold.
11 . The method of claim 1 , wherein generating the one or more pseudo-labels comprises:
determining the plurality of keypoint estimates are greater than a total score threshold; and
selecting the plurality of keypoint estimates as the one or more pseudo-labels based on the plurality of keypoint estimates being greater than the total score threshold.
12 . The method of claim 1 , wherein the one or more parameters include weights of the machine learning model.
13 . An apparatus for performing online adaptation of one or more machine learning models, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
obtain features extracted from a first image by a machine learning model during inference;
determine, using the machine learning model based on the features during inference, at least one of a plurality of keypoint estimates in the first image or a bounding region estimate associated with an object in the first image;
generate one or more pseudo-labels based on at least one of the plurality of keypoint estimates or the bounding region estimate;
determine at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the bounding region estimate;
adapt, based on the at least one self-supervised loss, one or more parameters of the machine learning model; and
generate, using the machine learning model with the adapted one or more parameters, a segmentation mask for the first image or a second image;
wherein the features are extracted from the first image by a feature extraction engine of the machine learning model; and
wherein adapting, based on the at least one self-supervised loss, the one or more parameters of the machine learning model comprises adapting one or more parameters of the feature extraction engine.
14 . The apparatus of claim 13 , wherein the at least one processor is configured to:
process, using the machine learning model with the adapted one or more parameters, the first image or a second image to generate updated features representing the first image or to generate features representing the second image.
15 . The apparatus of claim 14 , wherein the at least one processor is configured to generate the segmentation mask based on processing the updated features representing the first image or the features representing the second image.
16 . The apparatus of claim 13 , wherein the at least one processor is configured to determine the plurality of keypoint estimates in the first image using a keypoint detection engine of the machine learning model.
17 . The apparatus of claim 13 , wherein the at least one processor is configured to determine the bounding region estimate in the first image using an object detection engine of the machine learning model.
18 . The apparatus of claim 13 , wherein the at least one processor is configured to generate the segmentation mask using a segmentation mask engine of the machine learning model.
19 . The apparatus of claim 13 , wherein the plurality of keypoint estimates includes at least one heat map indicating probabilities that portions of the first image correspond to a particular keypoint.
20 . The apparatus of claim 19 , wherein the at least one heat map includes a plurality of heat maps, each heat map of the plurality of heat maps being associated with a respective keypoint of a plurality of keypoints.
21 . The apparatus of claim 13 , wherein the plurality of keypoint estimates includes coordinates associated with a plurality of keypoints.
22 . The apparatus of claim 13 , wherein, to generate the one or more pseudo-labels, the at least one processor is configured to:
determine one or more keypoint estimates from the plurality of keypoint estimates that are greater than an individual keypoint score threshold; and
select the one or more keypoint estimates as the one or more pseudo-labels based on the one or more keypoint estimates being greater than the individual keypoint score threshold.
23 . The apparatus of claim 13 , wherein, to generate the one or more pseudo-labels, the at least one processor is configured to:
determine the plurality of keypoint estimates are greater than a total score threshold; and
select the plurality of keypoint estimates as the one or more pseudo-labels based on the plurality of keypoint estimates being greater than the total score threshold.
24 . The apparatus of claim 13 , wherein the one or more parameters include weights of the machine learning model.
25 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:
obtain features extracted from a first image by a machine learning model during inference;
determine, using the machine learning model based on the features during inference, at least one of a plurality of keypoint estimates in the first image or a bounding region estimate associated with an object in the first image;
generate one or more pseudo-labels based on at least one of the plurality of keypoint estimates or the bounding region estimate;
determine at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the bounding region estimate;
adapt, based on the at least one self-supervised loss, one or more parameters of the machine learning model; and
generate, using the machine learning model with the adapted one or more parameters, a segmentation mask for the first image or a second image;
wherein the features are extracted from the first image by a feature extraction engine of the machine learning model; and
wherein adapting, based on the at least one self-supervised loss, the one or more parameters of the machine learning model comprises adapting one or more parameters of the feature extraction engine.
26 . An apparatus for performing online adaptation of one or more machine learning models, the apparatus comprising:
means for obtaining features extracted from a first image by a machine learning model during inference;
means for determining, by the machine learning model based on the features during inference, at least one of a plurality of keypoint estimates in the first image or a bounding region estimate associated with an object in the first image;
means for generating one or more pseudo-labels based on at least one of the plurality of keypoint estimates or the bounding region estimate;
means for determining at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the bounding region estimate;
means for adapting, based on the at least one self-supervised loss, one or more parameters of the machine learning model; and
means for generating, using the machine learning model with the adapted one or more parameters, a segmentation mask for the first image or a second image;
wherein the features are extracted from the first image by a feature extraction engine of the machine learning model; and
wherein adapting, based on the at least one self-supervised loss, the one or more parameters of the machine learning model comprises adapting one or more parameters of the feature extraction engine.