Learning semantic segmentation models in the absence of a portion of class labels
Performing semantic segmentation in an absence of labels for one or more semantic classes is provided. One or more weak predictors are utilized to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes. The label proposals are merged with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset. A machine learning model is trained using the merged dataset. The machine learning model is utilized for performing semantic segmentation on image data.
1. A method for performing semantic segmentation in an absence of labels for one or more semantic classes, comprising:
utilizing one or more weak predictors to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes;
merging the label proposals with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset;
training a machine learning model using the merged dataset; and
utilizing the machine learning model for performing semantic segmentation on image data.
2. The method of claim 1 , wherein the one or more weak predictors include a feature-based predictor configured to generate the label proposals by detecting edges within a region of interest in the original dataset.
3. The method of claim 1 , wherein the one or more weak predictors include a transfer-based predictor configured to generate the label proposals by using a transfer model trained to predict classes of interest on a transfer information dataset separate from the original dataset.
4. The method of claim 1 , wherein the merging of the label proposals includes
assigning pixels of the original dataset to the unlabeled classes if so indicated by the label proposals; and
assigning remaining pixels of the original dataset to the labeled classes specified by the ground truth of the original dataset.
5. The method of claim 1 , wherein the training of the machine learning model utilizes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth.
6. The method of claim 1 , wherein the training of the machine learning model utilizes a modified loss function that includes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth plus additional components that target losses specific to one or more of the novel classes.
7. The method of claim 1 , wherein the semantic segmentation of the image data includes assigning of pixels of the image data to classes indicated in the ground truth as well as to the novel classes indicated in the label proposals.
8. A system for performing semantic segmentation in an absence of labels for one or more semantic classes, comprising:
one or more processors programmed to:
utilize one or more weak predictors to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes;
merge the label proposals with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset;
train a machine learning model using the merged dataset; and
utilize the machine learning model for performing semantic segmentation on image data.
9. The system of claim 8 , wherein the one or more weak predictors include a feature-based predictor configured to generate the label proposals by detecting edges within a region of interest in the original dataset.
10. The system of claim 8 , wherein the one or more weak predictors include a transfer-based predictor configured to generate the label proposals by using a transfer model trained to predict classes of interest on a transfer information dataset separate from the original dataset.
11. The system of claim 8 , wherein to merge the label proposals includes to:
assign pixels of the original dataset to the unlabeled classes if so indicated by the label proposals; and
assign remaining pixels of the original dataset to the labeled classes specified by the ground truth of the original dataset.
12. The system of claim 8 , wherein to train the machine learning model utilizes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth.
13. The system of claim 8 , wherein to train the machine learning model utilizes a modified loss function that includes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth plus additional components that target losses specific to one or more of the novel classes.
14. The system of claim 8 , wherein the semantic segmentation of the image data includes to assign of pixels of the image data to classes indicated in the ground truth as well as to the novel classes indicated in the label proposals.
15. A non-transitory computer-readable medium comprising instructions for performing semantic segmentation in an absence of labels for one or more semantic classes, that, when executed by one or more processors case the one or more processors to perform operations including to:
utilize one or more weak predictors to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes;
merge the label proposals with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset;
train a machine learning model using the merged dataset; and
utilize the machine learning model for performing semantic segmentation on image data.
16. The medium of claim 15 , wherein the one or more weak predictors include a feature-based predictor configured to generate the label proposals by detecting edges within a region of interest in the original dataset.
17. The medium of claim 15 , wherein the one or more weak predictors include a transfer-based predictor configured to generate the label proposals by using a transfer model trained to predict classes of interest on a transfer information dataset separate from the original dataset.
18. The medium of claim 15 , wherein to merge the label proposals includes to:
assign pixels of the original dataset to the unlabeled classes if so indicated by the label proposals; and
assign remaining pixels of the original dataset to the labeled classes specified by the ground truth of the original dataset.
19. The medium of claim 15 , wherein to train the machine learning model utilizes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth.
20. The medium of claim 15 , wherein to train the machine learning model utilizes a modified loss function that includes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth plus additional components that target losses specific to one or more of the novel classes.
21. The medium of claim 15 , wherein the semantic segmentation of the image data includes to assign of pixels of the image data to classes indicated in the ground truth as well as to the novel classes indicated in the label proposals.