PRIVACY PRESERVING VISUAL LOCALIZATION

1 . A method for performing privacy-preserving visual localization, by a computing device, using a two dimensional (2D) query image captured by a camera, comprising:

(a) determining a query segmentation map based on the query image, wherein each pixel of the query segmentation map is associated with one or more likelihoods that it belongs to one or more segmentation classes that are scene-specific and learned in a self-supervised manner;

(b) accessing a privacy preserved scene representation that includes labeled three dimensional (3D) representations of a scene selected from the one or more segmentation classes, the privacy preserved scene representation comprising one or more of:

(i) a 3D point cloud generated by Structure-from-Motion (SfM);

(ii) a neural implicit field including a neural radiance field (NeRF) and/or associated geometric, segmentation and/or feature fields; and

(iii) a Gaussian Splatting Feature Field (GSFF) including a plurality of 3D Gaussian primitives;

(d) determining a predicted pose based on a starting pose; and

(e) generating, from the predicted pose, a predicted segmentation map;

(f) refining the predicted pose after aligning the predicted segmentation map with the query segmentation map, each of the query segmentation map and the predicted segmentation map including for each pixel one or more likelihoods that it belongs to the one or more segmentation classes, respectively;

(g) repeating the generating and the refining until the query segmentation map and the predicted segmentation map converge to within a predefined convergence criterion;

(h) outputting the refined predicted pose of the query image captured by the camera using the predicted segmentation map at (g) that converged to within the predicted convergence criterion.

2 . The method of claim 1 wherein the privacy preserved scene representation is generated from a set of training images representing the scene using one or more of (i) Structure-from-Motion (SfM) (ii) Neural Radiance Fields (NeRFs) and (iii) Gaussian Splatting Feature Fields (GSFF).

3 . The method of claim 2 further comprising generating (i) a global descriptor of the input image, and (ii) global descriptors with pose information of the set of training images representing the scene, each global descriptor aggregating image features into a single descriptor.

4 . The method of claim 3 wherein the starting pose is predicted based on similarities between the global descriptor for the query image and the global descriptors for the set of training images.

5 . The method of claim 1 wherein the privacy preserved scene representation includes labeled three dimensional (3D) representations of the scene (i) with texture and/or fine details obscured and (ii) from which views of the scene may be rendered.

6 . The method of claim 1 , wherein, when the privacy-preserved scene representation corresponds to the 3D point cloud generated by SfM, the 3D point cloud includes 3D points labeled by the one or more segmentation classes, wherein the one or more segmentation classes are derived in a self-supervised manner using pixel correspondences.

7 . The method of claim 1 , wherein, when the privacy-preserved scene representation corresponds to the neural implicit field, the neural implicit field comprises:

a segmentation field module configured to provide segmentation information on the scene in the training images; and

a geometric field module configured to provide geometric information on the scene;

wherein the neural implicit field is trained using segmentation labels as supervision such that high-frequency texture details are suppressed, and an internal representation of the scene is privacy-preserving.

8 . The method of claim 7 , wherein the neural implicit field comprises a feature field module configured to provide feature information on the scene.

9 . The method of claim 1 , wherein, when the privacy-preserved scene representation corresponds to the GSFF, the GSFF comprises one or more 3D Gaussian primitives each having a center, a covariance, an opacity and a feature or segmentation label, wherein images are rendered by rasterizing the one or more 3D Gaussian primitives using depth ordering and alpha blending.

10 . The method of claim 2 , further comprising:

generating a global descriptor of the query image;

accessing global descriptors with pose information of the set of training images representing the scene; and

determining the starting pose based on similarities between the global descriptor for the query image and the global descriptors for the training images.

11 . The method of claim 10 , wherein:

the query segmentation map comprises segmentation heatmaps;

the global descriptor is generated by applying a pooling operator to the segmentation heatmaps; and

selecting the starting pose includes selecting k images based on similarities between the global descriptor for the query image and global descriptors of the k images, respectively.

12 . The method of claim 1 wherein the aligning is performed using all pixels of the query segmentation map or only a subset of pixels of the query segmentation map.

13 . The method of claim 1 wherein the one or more segmentation classes define a non-injective mapping from RGB pixels or features to labels such that regions of the scene having different texture details share a same segmentation label to suppress retrieval of sensitive visual details.

14 . The method of claim 1 , further comprising:

determining a location in the scene using the refined predicted pose output at (h); and

performing, with the computing device, one or more tasks using the determined location;

wherein the one or more tasks are tailored to the location and include one or more of an auditory response concerning the location, a delivery to the location, and navigation to the location.

15 . The method of claim 14 , wherein the computing device is one of a robot and a virtual assistant.

16 . A system that performs visual localization using a two dimensional (2D) query image captured by a camera, the system comprising:

at least one processor;

at least one memory, wherein executable instructions stored in the at least one memory are configured to cause the at least one processor to:

(a) determine a query segmentation map based on the query image, wherein each pixel of the query segmentation map is associated with one or more likelihoods that it belongs to one or more segmentation classes that are scene-specific and learned in a self-supervised manner;

(b) access a privacy preserved scene representation that includes labeled three dimensional (3D) representations of a scene selected from one or more segmentation classes, the privacy preserved scene representation comprising one or more of;

(i) a 3D point cloud generated by Structure-from-Motion (SfM);

(ii) a neural implicit field including a neural radiance field (NeRF) and/or associated geometric, segmentation and/or feature fields; and

(iii) a Gaussian Splatting Feature Field (GSFF) including a plurality of 3D Gaussian primitives;

(d) determine a predicted pose based on a starting pose; and

(e) generate from the predicted pose a predicted segmentation map;

(f) refine the predicted pose after aligning the predicted segmentation map with the query segmentation map, each of the query segmentation map and the predicted segmentation map including for each pixel one or more likelihoods that it belongs to the one or more segmentation classes, respectively;

(g) repeat the generating and the refining until the query segmentation map and the predicted segmentation map converge to within a predefined convergence criterion;

(h) output the refined predicted pose of the query image captured by the camera using the predicted segmentation map at (g) that converged to within the predicted convergence criterion.

17 - 19 . (canceled)

20 . The system of claim 16 , wherein the executable instructions stored in the at least one memory are further configured to cause the at least one processor to:

determine a location in the scene using the refined predicted pose output at (h); and

perform one or more tasks using the determined location;

wherein the one or more tasks are tailored to the location and include one or more of an auditory response concerning the location, a delivery to the location, and navigation to the location.

21 . A training system for privacy preserving visual localization, comprising:

a pose module configured to receive training images captured using a camera and determine a six degrees of freedom (6 DoF) pose of the camera that captured each of the training images;

an encoder module including a segmentation module configured to determine at least one segmentation heatmap and at least one global descriptor based on an input image;

a scene-representation module configured to provide a privacy-preserved scene representation of a scene viewed in the training images, the scene-representation module being configured to implement one or more of:

a 3D point cloud generated by Structure-from-Motion having labeled 3D points;

a neural implicit field including a segmentation field module providing segmentation information, a geometric field module providing geometric information; and

a Gaussian Splatting Feature Field including 3D Gaussians each having a center, covariance, opacity, and feature or segmentation label; and

a training module configured to:

input the training images to the pose module;

determine prototype distributions or prototypes in a feature embedding space based on feature maps or volumetric features derived from the training images and the privacy-preserved scene representation; and

train at least the segmentation module and at least one of the pose module, the scene-representation module, the segmentation field module, and the geometric field module, by alternating between:

(i) updating a target distribution with parameters of the segmentation module fixed based on minimizing a first loss determined using a label distribution determined from the prototypes;

(ii) updating parameters of the segmentation module with the target distribution fixed based on minimizing a second loss that is different from the first loss; and

(iii) updating parameters of the segmentation module and/or the pose module based on a ranking loss using a global representation.

22 . The training system of claim 21 , wherein the second loss is a per-pixel cross-entropy loss between predicted segmentation heatmaps and pseudo-labels.

23 . The training system of claim 21 , wherein the training module is configured to train the segmentation module based on a first function based on feature vectors and prototype distributions during a first epoch of a predetermined number of epochs and a second function different from the first function during remaining epochs.

24 . The training system of claim 21 , wherein the training module is configured to train the pose module further based on minimizing a consistency loss, the consistency loss being determined based on at least one of:

labels assigned to keypoints in the training images based on distances to the prototype distributions; and

feature maps determined based on the training images.

25 . The training system of claim 21 , wherein the training module is further configured to train the segmentation module based on minimizing a contrastive loss determined based on the prototype distributions, feature maps, and concentrations of the prototype distributions.

26 . The training system of claim 21 , wherein the ranking loss comprises a multi-similarity loss applied to at least one global descriptor derived from the training images.

27 . The training system of claim 21 , wherein, when the scene-representation module implements the Gaussian Splatting Feature Field, the training module is further configured to:

cause a rasterization module to render a second feature map and a second segmentation map aligned with a first feature map and a first segmentation map extracted by the encoder module; and

train parameters of the encoder module based on at least one loss determined based on at least one of:

a difference between the first feature map and the second feature map; and

a difference between the first segmentation map and the second segmentation map.

28 . The training system of claim 27 , wherein the training module is further configured to:

apply spectral clustering on a Delaunay graph derived from a Gaussian cloud to produce a set of prototypes; and

generate labels for respective 3D Gaussians of the Gaussian cloud by assigning volumetric features to the prototypes.

29 . The training system of claim 21 , wherein, when the scene-representation module implements the neural implicit field with segmentation, the training module is configured to:

generate a first set of K prototypes based on features extracted from an input image;

generate a second set of K prototypes based on segmentation information, geometric information, and feature information produced by the segmentation field module, and the geometric field module;

align the first and second sets of K prototypes and determine segmentation targets based on mapping features to the prototypes based on similarities in the feature embedding space; and

jointly train the encoder module, the segmentation field module, and the geometric field module, using a cross-entropy loss based on the segmentation targets,

wherein K is an integer greater than zero and corresponds to a predetermined number of segmentation classes.

30 . A training method for privacy-preserving visual localization, the method comprising:

receiving, by a pose module executed by at least one processor, training images captured using a camera;

determining, by an encoder module including a segmentation module executed by the at least one processor, at least one segmentation heatmap and at least one global descriptor based on an input image;

providing, by a scene-representation module executed by the at least one processor, a privacy-preserved scene representation of a scene viewed in the training images, the scene-representation module implementing one or more of:

a 3D point cloud generated by Structure-from-Motion having labeled 3D points;

a neural implicit field including a segmentation field module providing segmentation information and a geometric field module providing geometric information; and

a Gaussian Splatting Feature Field including 3D Gaussians each having a center, covariance, opacity, and feature or segmentation label;

determining, by a training module executed by the at least one processor, prototype distributions or prototypes in a feature embedding space based on feature maps or volumetric features derived from the training images and the privacy-preserved scene representation; and

training, by the training module, at least the segmentation module and at least one of the pose module, the scene-representation module, the segmentation field module, and the geometric field module, by alternating between:

(i) updating a target distribution with parameters of the segmentation module fixed based on minimizing a first loss determined using a label distribution determined from the prototypes;

(ii) updating parameters of the segmentation module with the target distribution fixed based on minimizing a second loss that is different than the first loss; and

(iii) updating parameters of the segmentation module and/or the pose module based on a ranking loss using the global descriptor.

31 - 33 . (canceled)

34 . The training method of claim 30 , wherein determining the prototype distributions includes rejecting outlier feature vectors based on a distance threshold relative to a cluster center associated with one of the prototypes.

35 . The training method of claim 30 , further comprising storing, in a memory, intermediate prototype distributions generated during earlier epochs and reusing the intermediate prototype distributions for stabilizing later iterations of the training.

36 - 37 . (canceled)

38 . The training method of claim 30 , further comprising generating confidence values for respective segmentation classes based on (i) distances to respective prototypes or (ii) learning, and applying the confidence values during training and inference.

39 - 120 . (canceled)