IP Library › Granted Patent US 12,657,887
Granted Patent B2
US 12,657,887 · App. 18/452,780 · Granted Jun 16, 2026

Percentile-based pseudo-label selection for multi-label semi-supervised classification

Inventors: Junxiang Huang (Revere, MA); Alexander Huang (Fremont, CA); Beatriz Chinelato Guerra (San Antonio, TX); Yen-Yun Yu (Murray, UT)
Assignee: AUTODESK, INC.
G06V10/7753G06V10/764
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,657,887
App. No.
18/452,780
Granted
Jun 16, 2026
Kind
B2
Abstract

A method and system provide for augmenting a photograph. An unlabeled photograph is obtained. A weakly augmented photograph and a strongly augmented photograph are obtained from the unlabeled photograph based on different types of data augmentation methods. The weakly augmented photograph is processed through a model to generate multiple weakly augmented photograph class predictions (with assigned probabilities). The multiple weakly augmented photograph class predictions are converted into positive pseudo-labels (indicating a presence of a class) or negative pseudo-labels (indicating absence of a class) using different fixed percentile thresholds. The strongly augmented photograph is processed through the model to generate a strongly augmented photograph class prediction. The model is trained to make the strongly augmented photograph label prediction match the positive pseudo-label via a cross-entropy loss. The trained model is then utilized to label the unlabeled photograph with multiple labels.

Claims (67)

1 . A computer-implemented method for augmenting a photograph, comprising:

(a) obtaining an unlabeled photograph;

(b) obtaining a weakly augmented photograph based on the unlabeled photograph, wherein the weakly augmented photograph is based on a first type of data augmentation;

(c) obtaining a strongly augmented photograph based on the unlabeled photograph, wherein:

(i) the strongly augmented photograph is based on a second type of data augmentation; and

(ii) the strongly augmented photograph is more strongly augmented compared to that of the weakly augmented photograph;

(d) processing the weakly augmented photograph through a model to generate multiple weakly augmented photograph class predictions, wherein each weakly augmented photograph class prediction is assigned a probability by the model;

(e) converting the multiple weakly augmented photograph class predictions into positive pseudo-labels or negative pseudo-labels, wherein:

(i) a first class prediction, of the weakly augmented photograph class predictions, is converted to the positive pseudo-label when the probability for the first class prediction is above a positive score threshold, wherein the positive score threshold comprises a fixed percentile threshold that unifies fixed and variable score thresholds, and wherein the positive pseudo-label indicates a presence of a class;

(ii) a second class prediction, of the weakly augmented photograph class predictions, is converted to the negative pseudo-label when the probability for the second class prediction is below a negative score threshold, wherein the negative score threshold comprises a fixed percentile threshold that unifies fixed and variable score thresholds, and wherein the negative pseudo-label indicates an absence of a class; and

(iii) a third class prediction, of the weakly augmented photograph label predictions, is disregarded when the probability for the third class prediction is in between the positive score threshold and the negative score threshold;

(f) processing the strongly augmented photograph through the model to generate a strongly augmented photograph class prediction;

(g) training the model to make the strongly augmented photograph label prediction match the positive pseudo-label via a cross-entropy loss, wherein the training is based on the positive pseudo-labels and the negative pseudo-labels; and

(h) utilizing the model to label the unlabeled photograph with multiple labels.

2 . The computer-implemented method of claim 1 , wherein the photograph is of a construction project.

3 . The computer-implemented method of claim 1 , wherein the fixed percentile threshold is based on a ground truth ratio in labeled data that avoids having an incorrectly high ratio of the positive pseudo-labels.

4 . The computer-implemented method of claim 1 , wherein the training the model comprises:

determining a gap between the positive pseudo-label and the negative pseudo-label;

when the gap exceeds a threshold gap value, determining an unlabeled loss weight for the gap, wherein the unlabeled weight loss reflects a learning status for a class; and

updating the model based on the unlabeled loss weight, wherein the unlabeled weight loss indicates a performance level for converting to the positive pseudo-label, and wherein the unlabeled weight loss enables positive pseudo-labels of different learning difficulties to be involved at different times.

5 . The computer-implemented method of claim 4 , further comprising:

determining a total loss for converting to the positive pseudo-label as a weighted combination of supervised learning and the unlabeled loss weight, wherein the total loss is used during the training.

6 . The computer-implemented method of claim 4 , wherein:

as the training progresses, the positive score threshold increases and the negative score threshold decreases thereby increasing the gap.

7 . The computer-implemented method of claim 1 , further comprising:

empirically validating the augmenting using a public data set.

8 . The computer-implemented method of claim 1 , further comprising:

tracking progress of a construction project based on the labeled photograph.

9 . The computer-implemented method of claim 1 , further comprising:

modifying real world construction based on the labeled photograph.

10 . The computer-implemented method of claim 1 , further comprising:

enabling collaboration amongst multiple stakeholders in a construction project based on the labeled photograph.

11 . A computer-implemented system for augmenting a photograph, comprising:

(a) a computer having a memory;

(b) a processor executing on the computer;

(c) the memory storing a set of instructions, wherein the set of instructions, when executed by the processor cause the processor to perform operations comprising:

(i) obtaining an unlabeled photograph;

(ii) obtaining a weakly augmented photograph based on the unlabeled photograph, wherein the weakly augmented photograph is based on a first type of data augmentation;

(iii) obtaining a strongly augmented photograph based on the unlabeled photograph, wherein:

(1) the strongly augmented photograph is based on a second type of data augmentation; and

(2) the strongly augmented photograph is more strongly augmented compared to that of the weakly augmented photograph;

(iv) processing the weakly augmented photograph through a model to generate multiple weakly augmented photograph class predictions, wherein each weakly augmented photograph class prediction is assigned a probability by the model;

(v) converting the multiple weakly augmented photograph class predictions into positive pseudo-labels or negative pseudo-labels, wherein:

(1) a first class prediction, of the weakly augmented photograph class predictions, is converted to the positive pseudo-label when the probability for the first class prediction is above a positive score threshold, wherein the positive score threshold comprises a fixed percentile threshold that unifies fixed and variable score thresholds, and wherein the positive pseudo-label indicates a presence of a class;

(2) a second class prediction, of the weakly augmented photograph class predictions, is converted to the negative pseudo-label when the probability for the second class prediction is below a negative score threshold, wherein the negative score threshold comprises a fixed percentile threshold that unifies fixed and variable score thresholds, and wherein the negative pseudo-label indicates an absence of a class; and

(3) a third class prediction, of the weakly augmented photograph label predictions, is disregarded when the probability for the third class prediction is in between the positive score threshold and the negative score threshold;

(vi) processing the strongly augmented photograph through the model to generate a strongly augmented photograph class prediction;

(vii) training the model to make the strongly augmented photograph label prediction match the positive pseudo-label via a cross-entropy loss, wherein the training is based on the positive pseudo-labels and the negative pseudo-labels; and

(viii) utilizing the model to label the unlabeled photograph with multiple labels.

12 . The computer-implemented system of claim 11 , wherein the photograph is of a construction project.

13 . The computer-implemented system of claim 11 , wherein the fixed percentile threshold is based on a ground truth ratio in labeled data that avoids having an incorrectly high ratio of the positive pseudo-labels.

14 . The computer-implemented system of claim 11 , wherein the operation training the model comprises:

determining a gap between the positive pseudo-label and the negative pseudo-label;

when the gap exceeds a threshold gap value, determining an unlabeled loss weight for the gap, wherein the unlabeled weight loss reflects a learning status for a class; and

updating the model based on the unlabeled loss weight, wherein the unlabeled weight loss indicates a performance level for converting to the positive pseudo-label, and wherein the unlabeled weight loss enables positive pseudo-labels of different learning difficulties to be involved at different times.

15 . The computer-implemented system of claim 14 , wherein the operations further comprise:

determining a total loss for converting to the positive pseudo-label as a weighted combination of supervised learning and the unlabeled loss weight, wherein the total loss is used during the training.

16 . The computer-implemented system of claim 14 , wherein:

as the training progresses, the positive score threshold increases and the negative score threshold decreases thereby increasing the gap.

17 . The computer-implemented system of claim 11 , wherein the operations further comprise:

empirically validating the augmenting using a public data set.

18 . The computer-implemented system of claim 11 , wherein the operations further comprise:

tracking progress of a construction project based on the labeled photograph.

19 . The computer-implemented system of claim 11 , wherein the operations further comprise:

modifying real world construction based on the labeled photograph.

20 . The computer-implemented system of claim 11 , wherein the operations further comprise:

enabling collaboration amongst multiple stakeholders in a construction project based on the labeled photograph.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 21, 2023
From: HUANG, JUNXIANG; HUANG, ALEXANDER; GUERRA, BEATRIZ CHINELATO; YU, YEN-YUN
To: AUTODESK, INC.
Reel/Frame 064650/0325 →
Continuity (2)
Provisional Application 63373588 · Aug 26, 2022
Related Publication 20240071056A1 · Feb 29, 2024
References Cited (48)
US 20220156591A1 · Li · 2022 [cited by examiner]
US 20220392051A1 · Hur · 2022 [cited by examiner]
US 20230053716A1 · Weinzaepfel · 2023 [cited by examiner]
US 20230154167A1 · Li · 2023 [cited by examiner]
US 20240347175A1 · Wang · 2024 [cited by examiner]
Extended European Search Report dated Dec. 12, 2023 for European Patent Application No. 23193288.0. [cited by applicant]
Sohn, K., et al., “FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence”, arxiv.org, Cornell University Library, Cornell University, Jan. 2020, XP081582899, pp. 1-14. [cited by applicant]
Cascante-Bonilla, P., et al., “Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning”, arxiv.org, Cornell University Library, Cornell University, Dec. 2020, XP081834881, pp. 1-13. [cited by applicant]
Shenoy, A., “Pseudo-Labeling to deal with small datasets—What, Why & How? Towards Data Science”, Dec. 2019, XP093106564, https://towardsdatascience.com/pseudo-labeling-to-deal-with-smalldatasets-what-why-how-fd6f903213a… [cited by applicant]
European Decision to Grant dated Aug. 28, 2025 for European Patent Application No. 23193288.0. [cited by applicant]
European Communication under Rule 71 (3) EPC dated Mar. 24, 2025 for European Patent Application No. 23193288.0. [cited by applicant]
Tarvainen, A., et al., “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results”, Advances in Neural Information Processing Systems, 2017, pp. 1-10. [cited by applicant]
Zhang, B., et al. “FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling”, Advances in Neural Information Processing Systems, 2021, pp. 8408-18419, vol. 34, No. 1. [cited by applicant]
Gong, C., et al., “Multi-modal Curriculum Learning for Semi-supervised Image Classification”, IEEE Transactions on Image Processing, 2016, pp. 3249-3260, vol. 25, No. 7. [cited by applicant]
Xing, C. et al., “Distance-Based Learning From Errors for Confidence Calibration”, 8th International Conference on Learning Representations, ICLR 2020, OpenReview.net, 2020. [cited by applicant]
Raffel C., et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, Journal of Machine Learning Research, 2020, pp. 1-67, vol. 21, No. 140. [cited by applicant]
Gil, D., et al., “Classification of Images from Construction Sites Using a Deep-Learning Algorithm”, Proceedings of the International Symposium on Automation and Robotics in Construction, 2018, pp. 1-6, vol. 35. [cited by applicant]
Berthelot, D., et al., “MixMatch: A Holistic Approach to Semi-Supervised Learning”, 33rd Conference on Neural Information Processing Systems, 2019, pp. 1-11. [cited by applicant]
Mahajan, D., et al., “Exploring the Limits of Weakly Supervised Pretraining”, Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 181-196. [cited by applicant]
Lee, D-H., “Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks”, CML 2013 Workshop : Challenges in Representation Learning (WREPL), 2013, pp. 1-6, vol. 3. [cited by applicant]
Arazo, E., et al., “Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning”, 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. [cited by applicant]
Liu, F., et al., “Self-supervised Mean Teacher for Semi-supervised Chest X-ray Classification”, Machine Learning in Medical Imaging, 12th International Workshop, Sep. 2021, pp. 426-436. [cited by applicant]
Hacohen, G., et al., “On the Power of Curriculum Learning in Training Deep Networks”, International Conference on Machine Learning, PMLR, 2019, pp. 1-13. [cited by applicant]
Tokunaga, H., et al., “Negative Pseudo Labeling using Class Proportion for Semantic Segmentation in Pathology”, Computer Vision—ECCV 2020, pp. 430-446. [cited by applicant]
Elezi, I., et al., “Not All Labels Are Equal: Rationalizing the Labeling Costs for Training Object Detectio”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14492-14501. [cited by applicant]
Yalniz, I.Z., et al., “Billion-scale semi-supervised learning for image classification”, ArXiv, abs/1905.00546, 2019, pp. 1-12. [cited by applicant]
Hestness, J., et al., “Deep Learning Scaling is Predictable, Empirically”, CoRR, abs/1712.00409, 2017, pp. 1-19. [cited by applicant]
Sohn, K., et al., “Fixmatch: Simplifying semi-supervised learning with consistency and confidence”, 34th Conference on Neural Information Processing Systems, 2020, pp. 596-608. [cited by applicant]
Kingma, D.P., et al., “Adam: A method for stochastic optimization”, arXiv preprint arXiv:1412.6980, 2014, pp. 1-15. [cited by applicant]
Van Der Maaten, L., et al., “Visualizing data using t-sne”, Journal of Machine Learning Research, 2008, pp. 2579-26059, vol. 11, 2008. [cited by applicant]
Everingham, M., et al., “The PASCAL Visual Object Classes (VOC) Challenge”, International Journal of Computer Vision, Jun. 2010, pp. 303-338, vol. 88, No. 2. [cited by applicant]
Rizve, M.N., et al., “IIN Defense of Pseudo-Labeling: an Uncertainty-Aware Pseudo-Label Selection Framework for Semi-Supervised Learning”, International Conference on Learning Representations, 2021. arXiv preprint arXiv… [cited by applicant]
Sajjadi, M.S.M., et al., “Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning”, 30th Conference on Neural Information Processing Systems (NIPS 2016), pp. 1-9. [cited by applicant]
Xie, Q., et al., “Unsupervised Data Augmentation for Consistency Training”, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, Dec. 2020, pp. 1-13. [cited by applicant]
Laine, S., et al., et al., “Temporal Ensembling for Semi-Supervised Learning”, ArXiv, abs/1610.02242, 2017, pp. 1-13. [cited by applicant]
Ridnik, T., et al., “Asymmetric Loss for Multi-Label Classification”, Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 82-91. [cited by applicant]
Ridnik, T., et al., “ML-Decoder: Scalable and Versatile Classification Head”, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 32-41. [cited by applicant]
Chen, T., et al., “A Simple Framework for Contrastive Learning of Visual Representations”, International Conference on Machine Learning, 2020, pp. 1597-1607. [cited by applicant]
Lin, T-Y., et al., “Microsoft COCO: Common Objects in Context”, European Conference on Computer Vision, 2014, pp. 740-755. [cited by applicant]
Yang, X., et al., A Survey on Deep Semi-Supervised Learning, IEEE Transactions on Knowledge and Data Engineering, 2023, pp. 8934-8954, vol. 35, No. 9. [cited by applicant]
Zhu, X., Semi-supervised Learning Literature Survey, Technical Report, Sep. 2005, pp. 1-39. [cited by applicant]
Wang, X., et al., “ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases”, 2017 IEEE Conference on Computer Vision and Pattern Rec… [cited by applicant]
Wu, X., et al., “When Do Curricula Work?”, 9th International Conference on Learning Representations, May 2021, pp. 1-23. [cited by applicant]
Lecun, Y., et al., “Gradient-based learning applied to document recognition”, Proceedings of the IEEE, 1998, 86(11), pp. 2278-2324. [cited by applicant]
Xu, Y., et al., “Dash: Semi-Supervised Learning with Dynamic Thresholding”, Proceedings of the 38th International Conference on Machine Learning, Jul. 2021, pp. 11525-11536, vol. 139 of Proceedings of Machine Learning R… [cited by applicant]
Wang, Y., et al., “FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning”, CoRR, abs/ arXiv preprint arXiv:2205.07246, 2022, pp. 1-18. [cited by applicant]
Wang, Y., et al., “Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels” ArXiv, abs/2203.03884, 2022. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-13. [cited by applicant]
Zheng, Z., et al., “Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation” International Journal of Computer Vision, 2021, pp. 1106-1120, vol. 129, No. 4. [cited by applicant]