System and method for relocalization and scene recognition
A system configured to improve the operations associated with generating virtual representations of physical environments to recognize the physical environments and/or relocalize within the virtual representations in a substantially real time system. In some cases, the system may use a first pre-training phase of descriptors and/or split nodes of regression forests using features common across various scenes to learn general image appearance, and a second training phase of descriptors and/or leaf nodes of regression forests to learn scene specific features. The system may align the features using an orientation vector, correct for camera perspective and lens distortion of the features as well as learn robust illumination invariant features from real and synthetic data.
1. A system comprising:
a display for presenting a virtual environment to a user;
one or more image components for capturing image data associated with a physical environment surrounding the user;
one or more processors;
non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform operations including:
determining a set of features using training red-green-blue depth (RGBD) data associated with a type of scene;
determining one or more feature descriptors associated with the set of features;
determining a 3D gravity vector based at least in part on orientation data associated with the training RGBD data;
gravity aligning individual feature descriptors of the one or more feature descriptors by projecting the 3D gravity vector into a 2D image space;
generating synthetic training data based at least in part on the training RGBD data and at least one of the set of features;
determining one or more candidate features based at least in part on the synthetic training data; and
selecting at least one selected candidate feature of the one or more candidate features to include in the set of features based at least in part on a result of comparing feature descriptors of the one or more candidate features to the individual feature descriptors of the one or more feature descriptors.
2. The system as recited in claim 1 , wherein:
the training RGBD data includes images having a first viewpoint of an environment; and
the synthetic training data includes at least a second viewpoint of the environment, the second viewpoint rendered by the system.
3. The system as recited in claim 1 , wherein:
the training RGBD data includes images having at least one object within an environment or scene; and
the synthetic training data includes rendering the at least one object under different illumination effects.
4. The system as recited in claim 1 , wherein:
the training RGBD data includes images having an object within an environment, the objects having a first set of material characteristics; and
the synthetic training data includes rendering the objects with a second set of material characteristics, the second set of material characteristics different than the first set of material characteristics.
5. The system as recited in claim 4 , wherein the synthetic training data includes rendering the objects with a third set of material characteristics, the third set of material characteristics different than the first set of material characteristics and the second set of material characteristics.
6. The system as recited in claim 5 , wherein the operations are performed during an offline mode.
7. The system as recited in claim 5 , wherein determining the one or more candidate features is based at least in part on color data associated with the training RGBD data.
8. The system as recited in claim 4 , wherein training the plurality of the features includes:
discarding feature descriptors of the one or more feature descriptors based at least in part on depth data associated with the training RGBD data;
generating pixel comparison pairs for the individual feature descriptors of the one or more feature descriptors; and
generating a locality-sensitive-hashing index for the individual feature descriptors of the one or more feature descriptors.
9. The system as recited in claim 8 , further comprising an inertial measure unit to capture the orientation data.
10. The method as recited in claim 1 , wherein the image component is configured to capture the image data from substantially a same perspective as a perspective of the virtual environment.
11. The system as recited in claim 1 , wherein:
the training RGBD data includes images having an object within an environment, the objects having a first set of material characteristics; and
the synthetic training data includes rendering the objects with a second set of material characteristics, the second set of material characteristics different than the first set of material characteristics.
12. A method comprising:
receiving training red-green-blue-depth data;
detecting one or more features within the training RGBD data;
determining one or more feature descriptors;
discarding feature descriptors of the one or more feature descriptors based at least in part on depth data associated with the training RGBD data;
determining a 3D orientation vector based at least in part on orientation data associated with the training RGBD data;
aligning individual feature descriptors of the one or more feature descriptors by projecting the 3D orientation vector into a 2D image space;
generating pixel comparison pairs for the individual feature descriptors of the one or more feature descriptors; and
generating a locality-sensitive-hashing index for the individual feature descriptors of the one or more feature descriptors.
13. The method as recited in claim 12 , wherein the training RGBD data includes a plurality of frames, each frame having an associated pose data, gravity vector, depth data, and color data.
14. The method as recited in claim 12 , wherein the feature descriptors include an orientation descriptor.
15. The method as recited in claim 12 , wherein the 3D orientation vector is at least one of a gravity vector or a magnetic vector.
16. The method as recited in claim 12 , further comprising:
generating synthetic training data based at least in part on the training RGBD data and at least one of the set of models;
determining one or more candidate feature descriptors from the synthetic training data;
aligning individual candidate feature descriptors of the one or more candidate feature descriptors by projecting the 3D orientation vector into the 2D image space; and
selecting a first candidate feature descriptors to include in the locality-sensitive-hashing index based at least in part on a result of comparing the first candidate feature descriptors to the individual feature descriptors of the one or more feature descriptors.
17. A method comprising:
determining a set of features using training red-green-blue depth (RGBD) data associated with a type of scene;
generating synthetic training data based at least in part on the training RGBD data and at least one of the set of features;
determining one or more candidate features based at least in part on the synthetic training data;
determining a 3D gravity vector based at least in part on orientation data associated with the training RGBD data;
gravity aligning the candidate features based at least in part on the 3D gravity vector; and
selecting at least one of the one or more gravity aligned candidate features to include in the set of features based at least in part on a result of comparing individual candidate models of the one or more candidate features to individual ones of the set of models.
18. The method as recited in claim 17 , wherein:
the training RGBD data includes images having a first viewpoint of an environment; and
the synthetic training data includes at least a second viewpoint of the environment.
19. The method as recited in claim 17 , wherein:
the training RGBD data includes images having at least one object within an environment or scene; and
the synthetic training data includes rendering the at least one object under different illumination effects.
20. The system as recited in claim 10 , wherein the synthetic training data includes rendering the objects with a third set of material characteristics, the third set of material characteristics different than the first set of material characteristics and the second set of material characteristics.