IP Library › Granted Patent US 12,462,057
Granted Patent B2
US 12,462,057 · App. 18/598,198 · Granted Nov 4, 2025

Image feature matching with formal privacy guarantees

Inventors: Francesco Pittaluga (Los Angeles, CA); Bingbing Zhuang (Santa Clara, CA); Xiang Yu (Mountain View, CA)
Assignee: NEC Corporation
G06F21/6227G06V10/751
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,462,057
App. No.
18/598,198
Granted
Nov 4, 2025
Kind
B2
Abstract

Systems and methods are provided for privacy-preserving image feature matching in computer vision applications, including receiving a raw image descriptor, and perturbing the raw image descriptor using a subset selection mechanism to generate a perturbed descriptor set that includes the raw image descriptor and additional descriptors. Each descriptor in the perturbed descriptor set is replaced with its nearest neighbor in a predefined descriptor database to reduce the output domain size of the subset selection mechanism. Local differential privacy (LDP) protocols are employed to further perturb the descriptor set, ensuring formal privacy guarantees, and the perturbed descriptor set is matched against a second set of descriptors for image feature matching.

Claims (36)

1 . A method for privacy-preserving image feature matching in computer vision applications, comprising:

perturbing a raw image descriptor using a subset selection mechanism to generate a perturbed descriptor set that includes the raw image descriptor and additional descriptors;

replacing each descriptor in the perturbed descriptor set with its nearest neighbor in a predefined descriptor database to reduce an output domain size of the subset selection mechanism;

employing local differential privacy (LDP) protocols to further perturb the descriptor set, ensuring formal privacy guarantees; and

matching the perturbed descriptor set against a second set of descriptors for image feature matching.

2 . The method of claim 1 , wherein the subset selection mechanism employs a hybrid approach for perturbation, combining real-world descriptors with randomly generated descriptors.

3 . The method of claim 1 , where the predefined descriptor database is created from a comparatively large public database of images using clustering techniques.

4 . The method of claim 1 , wherein the LDP protocol includes applying a ω-Subset Mechanism for descriptor perturbation.

5 . The method of claim 1 , further comprising matching the perturbed descriptor set by computing point-to-subspace and subspace-to-subspace distances.

6 . The method of claim 1 , further comprising application of adversarial affine subspace embeddings for initial perturbation of the raw image descriptor.

7 . The method of claim 1 , where the matching includes photometric matching and geometric verification.

8 . A system for privacy-preserving image feature matching in computer vision applications, comprising:

a processor operatively coupled to a non-transitory computer-readable storage medium, the processor configured for:

perturbing a raw image descriptor using a subset selection mechanism to

generating a perturbed descriptor set that includes the raw image descriptor and additional descriptors;

replacing each descriptor in the perturbed descriptor set with its nearest neighbor in a predefined descriptor database to reduce an output domain size of the subset selection mechanism;

employing local differential privacy (LDP) protocols to further perturb the descriptor set, ensuring formal privacy guarantees; and

matching the perturbed descriptor set against a second set of descriptors for image feature matching.

9 . The system of claim 8 , wherein the subset selection mechanism employs a hybrid approach for perturbation, combining real-world descriptors with randomly generated descriptors.

10 . The system of claim 8 , where the predefined descriptor database is created from a comparatively large public database of images using clustering techniques.

11 . The system of claim 8 , wherein the LDP protocol includes applying a ω-Subset Mechanism for descriptor perturbation.

12 . The system of claim 8 , further comprising matching the perturbed descriptor set by computing point-to-subspace and subspace-to-subspace distances.

13 . The system of claim 8 , further comprising application of adversarial affine subspace embeddings for initial perturbation of the raw image descriptor.

14 . The system of claim 8 , where the matching includes photometric matching and geometric verification.

15 . A computer program product for disentangled data generation, the computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for privacy-preserving image feature matching in computer vision applications, comprising:

perturbing a raw image descriptor using a subset selection mechanism to

generate a perturbed descriptor set that includes the raw image descriptor and additional descriptors;

replacing each descriptor in the perturbed descriptor set with its nearest neighbor

in a predefined descriptor database to reduce an output domain size of the subset selection mechanism;

employing local differential privacy (LDP) protocols to further perturb the descriptor set, ensuring formal privacy guarantees; and

matching the perturbed descriptor set against a second set of descriptors for image feature matching.

16 . The computer-readable storage medium of claim 15 , wherein the subset selection mechanism employs a hybrid approach for perturbation, combining real-world descriptors with randomly generated descriptors.

17 . The computer-readable storage medium of claim 15 , where the predefined descriptor database is created from a comparatively large public database of images using clustering techniques.

18 . The computer-readable storage medium of claim 15 , wherein the LDP protocol includes applying a ω-Subset Mechanism for descriptor perturbation.

19 . The computer-readable storage medium of claim 15 , further comprising matching the perturbed descriptor set by computing point-to-subspace and subspace-to-subspace distances.

20 . The computer-readable storage medium of claim 15 , further comprising application of adversarial affine subspace embeddings for initial perturbation of the raw image descriptor.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 30, 2025
From: NEC LABORATORIES AMERICA, INC.
To: NEC CORPORATION
Reel/Frame 072420/0343 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 7, 2024
From: PITTALUGA, FRANCESCO; ZHUANG, BINGBING; YU, XIANG
To: NEC LABORATORIES AMERICA, INC.
Reel/Frame 066680/0812 →
Continuity (5)
Provisional Application 63460051 · Apr 18, 2023
Provisional Application 63460049 · Apr 18, 2023
Provisional Application 63460052 · Apr 18, 2023
Provisional Application 63488813 · Mar 7, 2023
Related Publication 20240303365A1 · Sep 12, 2024
References Cited (58)
US 9043250B2 · Ling · 2015 [cited by examiner]
US 20130163874A1 · Shechtman · 2013 [cited by examiner]
US 20200394320A1 · Bernau · 2020 [cited by examiner]
US 20220327438A1 · Bach · 2022 [cited by examiner]
US 20230058906A1 · Jansen · 2023 [cited by examiner]
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (Jul. 4, 2017). Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6), 1452-1464. [cited by applicant]
Nister, D., & Stewenius, H. (Jun. 17. 2006). Scalable recognition with a vocabulary tree. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) (vol. 2, pp. 2161-2168). Ieee. [cited by applicant]
Wu, Z., Wang, Z., Wang, Z., & Jin, H. (Sep. 8, 2018). Towards privacy-preserving visual recognition via adversarial training: A pilot study. In Proceedings of the European conference on computer vision (ECCV) (pp. 606-6… [cited by applicant]
The Visual Localization Benchmark. https://www.visuallocalization.net (Retrieved on Mar. 4, 2024). [cited by applicant]
Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., & Szeliski, R. (Oct. 1, 2011). Building rome in a day. Communications of the ACM, 54(10), 105-112. [cited by applicant]
Balntas, V., Riba, E., Ponsa, D., & Mikolajczyk, K. (Sep. 19, 2016). Learning local feature descriptors with triplets and shallow convolutional neural networks. In Bmvc (vol. 1, No. 2, p. 3). [cited by applicant]
Bishop, C. M. (Aug. 17, 2006). Pattern recognition and machine learning. Springer google scholar, 2, 5-43. [cited by applicant]
Cai, Z., Xiong, Z., Xu, H., Wang, P., Li, W., & Pan, Y. (Jul. 13, 2021). Generative adversarial networks: A survey toward private and secure applications. ACM Computing Surveys (CSUR), 54(6), 1-38. [cited by applicant]
Calonder, M., Lepetit, V., Ozuysal, M., Trzcinski, T., Strecha, C., & Fua, P. (Nov. 15, 2011). BRIEF: Computing a local binary descriptor very fast. IEEE transactions on pattern analysis and machine intelligence, 34(7),… [cited by applicant]
Chelani, K., Kahl, F., & Sattler, T. (Jun 20, 2021). How privacy-preserving are line clouds? recovering scene details from 3d lines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (… [cited by applicant]
Dalal, N., & Triggs, B. (Jun. 20, 2005). Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) (vol. 1, pp. 886-893). Ieee. [cited by applicant]
Dosovitskiy, A., & Brox, T. (Dec. 5, 2016). Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29. [cited by applicant]
Dosovitskiy, A., & Brox, T. (Jun. 26, 2016). Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4829-4837). [cited by applicant]
Dusmanu, M., Schonberger, J. L., Sinha, S. N., & Pollefeys, M. (Jun. 19, 2021). Privacy-preserving image features via adversarial affine subspace embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision … [cited by applicant]
Dwork, C. (Jul. 10, 2006). Differential privacy. In International colloquium on automata, languages, and programming (pp. 1-12). Berlin, Heidelberg: Springer Berlin Heidelberg. [cited by applicant]
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (Mar. 4, 2006). Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA… [cited by applicant]
Fischler, M. A., & Bolles, R. C. (Jun. 1, 1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381-395. [cited by applicant]
Geppert, M., Larsson, V., Speciale, P., Schönberger, J. L., & Pollefeys, M. (Aug. 23, 2020). Privacy preserving structure-from-motion. In Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, Proceedings, Pa… [cited by applicant]
Geppert, M., Larsson, V., Speciale, P., Schonberger, J. L., & Pollefeys, M. (Jun. 19, 2021). Privacy preserving localization and mapping from uncalibrated cameras. In Proceedings of the IEEE/CVF Conference on Computer V… [cited by applicant]
Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge university press. [cited by applicant]
Hassan, M. U., Rehmani, M. H., & Chen, J. (Sep. 27, 2019). Differential privacy techniques for cyber physical systems: a survey. IEEE Communications Surveys & Tutorials, 22(1), 746-789. [cited by applicant]
He, K., Lu, Y., & Sclaroff, S. (Jun. 18, 2018). Local descriptors optimized for average precision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 596-605). [cited by applicant]
Hsu, J., Gaboardi, M., Haeberlen, A., Khanna, S., Narayan, A., Pierce, B. C., & Roth, A. (Jul. 19, 2014). Differential privacy: An economic method for choosing epsilon. In 2014 IEEE 27th Computer Security Foundations Sy… [cited by applicant]
Announcing Azure Spatial Anchors for Collaborative, Cross-Platform Mixed Reality Apps. https://azure.microsoft.com/en-us/blog/announcing-azure-spatial-anchors-for-collaborative-cross-platform-mixed-reality-apps/ (Feb. 2… [cited by applicant]
Kato, H., & Harada, T. (Jun. 23, 2014). Image reconstruction from bag-of-visual-words. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 955-962). [cited by applicant]
Li, Z., & Snavely, N. (Jun. 18, 2018). Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2041-2050). [cited by applicant]
Long, J., Shelhamer, E., & Darrell, T. (Jun. 8, 2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440). [cited by applicant]
Lowe, D. G. (Jan. 5, 2004). Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60, 91-110. [cited by applicant]
Mahendran, A., & Vedaldi, A. (Jun. 8, 2015). Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5188-5196). [cited by applicant]
McPherson, R., Shokri, R., & Shmatikov, V. (Sep. 6, 2016). Defeating image obfuscation with deep learning. arXiv preprint arXiv:1609.00408. [cited by applicant]
Mishchuk, A., Mishkin, D., Radenovic, F., & Matas, J. (Dec. 4, 2017). Working hard to know your neighbor's margins: Local descriptor learning loss. Advances in neural information processing systems, 30. [cited by applicant]
Ng, T., Kim, H. J., Lee, V. T., DeTone, D., Yang, T. Y., Shen, T., . . . & Sweeney, C. (Jun. 19, 2022). NinjaDesc: content-concealing visual descriptors via adversarial learning. In Proceedings of the IEEE/CVF Conferenc… [cited by applicant]
Pittaluga, F., Koppal, S., & Chakrabarti, A. (Jan. 7, 2019). Learning privacy preserving encodings through adversarial training. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 791-799). IE… [cited by applicant]
Pittaluga, F., Koppal, S. J., Kang, S. B., & Sinha, S. N. (Jun. 16, 2019). Revealing scenes by inverting structure from motion reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re… [cited by applicant]
Revaud, J., Almazán, J., Rezende, R. S., & Souza, C. R. D. (Oct. 27, 2019). Learning with average precision: Training image retrieval with a listwise loss. In Proceedings of the IEEE/CVF International Conference on Comp… [cited by applicant]
Schonberger, J. L., & Frahm, J. M. (Jun. 26, 2016). Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4104-4113). [cited by applicant]
Schonberger, J. L., Hardmeier, H., Sattler, T., & Pollefeys, M. (Jul. 21, 2017). Comparative evaluation of hand-crafted and learned local features. In Proceedings of the IEEE conference on computer vision and pattern re… [cited by applicant]
Shibuya, M., Sumikura, S., & Sakurada, K. (Aug. 23, 2020). Privacy preserving visual SLAM. In Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, Proceedings, Part XXII 16 (pp. 102-118). Springer Internati… [cited by applicant]
Song, Z., Chen, W., Campbell, D., & Li, H. (Aug. 23, 2020). Deep novel view synthesis from colored 3D point clouds. In Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, Proceedings, Part XXIV 16 (pp. 1-1… [cited by applicant]
Speciale, P., Schonberger, J. L., Kang, S. B., Sinha, S. N., & Pollefeys, M. (Jun. 16, 2019). Privacy preserving image-based localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni… [cited by applicant]
Speciale, P., Schonberger, J. L., Sinha, S. N., & Pollefeys, M. (Oct. 27, 2019). Privacy preserving image queries for camera localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. … [cited by applicant]
Turk, M., & Pentland, A. (Jan. 1, 1991). Eigenfaces for recognition. Journal of cognitive neuroscience, 3(1), 71-86. [cited by applicant]
Vasiljevic, I., Chakrabarti, A., & Shakhnarovich, G. (May 30, 2017). Examining the impact of blur on recognition by convolutional networks. arXiv preprint arXiv:1611.05760. [cited by applicant]
Vondrick, C., Khosla, A., Malisiewicz, T., & Torralba, A. (Jun. 23, 2013). Hoggles: Visualizing object detection features. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-8). [cited by applicant]
Wu, Z., Wang, H., Wang, Z., Jin, H., & Wang, Z. (Sep. 28, 2020). Privacy-preserving deep action recognition: An adversarial learning framework and a new dataset. IEEE Transactions on Pattern Analysis and Machine Intelli… [cited by applicant]
Wang, T., Blocki, J., Li, N., & Jha, S. (Aug. 16, 2017). Locally differentially private protocols for frequency estimation. In 26th USENIX Security Symposium (USENIX Security 17) (pp. 729-745). [cited by applicant]
Wang, T., Zhang, X., Feng, J., & Yang, X. (Dec. 8, 2020). A comprehensive survey on local differential privacy toward data statistics and analysis. Sensors, 20(24), 7030. [cited by applicant]
Weinzaepfel, P., Jégou, H., & Pérez, P. (Jun. 20, 2011). Reconstructing an image from its local descriptors. In CVPR 2011 (pp. 337-344). IEEE. [cited by applicant]
Xiao, T., Tsai, Y. H., Sohn, K., Chandraker, M., & Yang, M. H. (Apr. 3, 2020). Adversarial learning of privacy-preserving and task-oriented representations. In Proceedings of the AAAI Conference on Artificial Intelligen… [cited by applicant]
Xiong, X., Liu, S., Li, D., Cai, Z., & Niu, X. (Oct. 8, 2020). A comprehensive survey on local differential privacy. Security and Communication Networks, 2020, 1-29. [cited by applicant]
Yang, M., Guo, T., Zhu, T., Tjuawinata, I., Zhao, J., & Lam, K. Y. (Dec. 30, 2023). Local differential privacy and its applications: A comprehensive survey. Computer Standards & Interfaces, 103827. [cited by applicant]
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., & Lipson, H. (Jun. 22, 2015). Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579. [cited by applicant]
Zeiler, M. D., & Fergus, R. (Aug. 14, 2014). Visualizing and understanding convolutional networks. In Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland, Sep. 6-12, 2014, Proceedings, Part I 13 (pp… [cited by applicant]