Deep neural network pose estimation system
A deep neural network provides real-time pose estimation by combining two custom deep neural networks, a location classifier and an ID classifier, with a pose estimation algorithm to achieve a 6D0F location of a fiducial marker. The locations may be further refined into subpixel coordinates using another deep neural network. The networks may be trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and/or extreme data augmentation. The deep neural network provides improved pose estimations particularly in challenging low-light, high-motion, and/or high-blur scenarios.
1. A system comprising:
a mixed reality device comprising a wearable headset and a handheld controller; a hardware computer processor;
a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the system to perform operations comprising:
accessing, from an imaging device, an image comprising a plurality of fiducial markers on a substrate;
applying, using a two-headed convolutional neural network, a plurality of stages of encoding to the image to generate an encoded image with reduced dimensionality;
applying a first neural network head of the two-headed convolutional neural network to the encoded image to determine two-dimensional locations of fiducial markers in the encoded image;
applying a second neural network head of the two-headed convolutional neural network to the encoded image to determine identifiers associated with respective fiducial markers detected in the encoded image;
determining, using a subpixel refinement component, a subpixel corner location based on the two-dimensional locations; and
determining a pose of the imaging device based at least on the subpixel corner location and the determined identifiers of the fiducial markers.
2. The system of claim 1 , wherein the pose of the imaging device is determined based on application of a perspective-n-point algorithm.
3. The system of claim 1 , wherein the pose is further determined based on one or more intrinsic parameters of the imaging device.
4. The system of claim 1 , wherein the operations further comprise:
outputting, from the first neural network head, a first plurality of image patches of the encoded image; and
determining, for each of the first plurality of image patches, based on application of a subpixel localization algorithm, a plurality of subpixel corners.
5. The system of claim 4 , wherein the subpixel corners indicate locations at a multiple of a resolution of the encoded image.
6. The system of claim 4 , wherein determining the pose of the imaging device is further based on the plurality of subpixel corners.
7. The system of claim 1 , wherein the fiducial markers are ChArUco markers.
8. The system of claim 1 , wherein the plurality of stages of encoding comprise one or more max-pooling layers.
9. The system of claim 8 , wherein spatial dimensionality of the image is reduced by a factor of eight in the encoded image.
10. The system of claim 1 , wherein a quantity of image patches is based on a maximum quantity of fiducial markers in the image.
11. The system of claim 1 , wherein the imaging device comprising a camera of a wearable augmented reality headset.
12. The system of claim 11 , wherein virtual content displayed by the wearable augmented reality headset is determined based at least partly on the pose of the imaging device.
13. The system of claim 1 , wherein the two-dimensional locations of fiducial markers in the encoded image comprise, for each of a plurality of fiducial marker locations on the substrate, probabilities of each of multiple fiducial markers being at the fiducial marker location.
14. The system of claim 13 , wherein each the plurality of fiducial marker locations are each different portions of the encoded image associated with the substrate.
15. The system of claim 1 , wherein the determined identifiers associated with fiducial markers in the encoded image comprise, for each of a plurality of fiducial marker locations on the substrate, probabilities of each of multiple identifiers being encoded at the fiducial marker location.
16. A system for determining pose of a camera, the system comprising:
an encoder comprising one or more processors and performing spatial downsampling of an image acquired from an augmented reality headset to generate a spatially downsampled image;
a two-headed convolutional neural network comprising:
a first head that detects two-dimensional locations of fiducial boards in the spatially downsampled image;
a second head that determines identifiers encoded at the two-dimensional locations of fiducial boards in the spatially downsampled image;
a subpixel refinement component that receives as input image patches of the spatially downsampled image and, for each image patch, provides as output a subpixel corner location determined based on the two-dimensional locations of fiducial boards in the spatially downsampled image; and
a pose estimation component that determines a pose augmented reality headset based at least on the two-dimensional locations and the subpixel corner locations.
17. The system of claim 16 , wherein the spatial downsampling comprises one or more pooling layers.
18. The system of claim 16 , further comprising:
a training component configured to train the two-headed convolutional neural network based on a plurality of video sequences wherein each frame of the video sequences undergoes a synthetic distortion.
19. The system of claim 18 , wherein synthetic distortions are selected randomly to achieve a desired proportion of each of a plurality of synthetic distortions.
20. The system of claim 19 , wherein the synthetic distortions comprise one or more of:
added Gaussian
noise; motion
blur;
Gaussian blur;
speckle noise;
brightness
rescale;
shadow or spotlight
effect; or homographic
transform.