Sensor fusion methods for augmented reality navigation
Embodiments of the present disclosure are directed to providing an Augmented Reality (AR) navigation display in a vehicle. More specifically, embodiments are directed to rendering AR indications of a navigation route over a camera video stream in perspective. According to one embodiment, visual tracking can be performed on features in the video data and camera pose, i.e., a matrix encapsulating position and orientation, can be determined for each frame of video based on both the visual tracking and navigation sensor data. These separately determined camera poses can then be merged or fused into a single camera pose that is more accurate and more stable and which can then be used in rendering more realistic AR route indicators onto the video of the real-world route captured by the camera.
1. A method for providing an Augmented Reality (AR) navigation display in a vehicle, the method comprising:
initializing, by a processor of a control system of the vehicle, a camera of the vehicle and one or more sensors of the vehicle, wherein initializing the camera and the one or more sensors comprises calculating an intrinsic camera matrix for the camera, calculating route world positions for a route being navigated by the vehicle, initializing the camera for capturing video data, and initializing the one or more sensors for capturing sensor data;
receiving, by the processor, the sensor data from the one or more sensors of the vehicle, the received sensor data comprising an indication of a current geographic location of the vehicle;
receiving, by the processor, the video data from the camera of the vehicle, the received video data comprising a plurality of frames with each frame representing a scene from a route being navigated by the vehicle;
determining, by the processor, a current fused camera pose for each frame of the received video data based on a combination of the received sensor data and visual tracking of features in the frames of the received video data, wherein determining the current fused camera pose for each frame of the received video data based on a combination of the received sensor data and the received video data comprises performing the visual tracking of the features in the frames of the received video data, determining a visual camera pose for the camera for each frame of the received video data based on the visual tracking, determining a sensor camera pose for the camera for each frame of the received video data based on the received sensor data, and combining the determined visual camera pose and the determined sensor camera pose into the determined current fused camera pose for each frame of the received video data based on a previous camera pose; and
rendering, by the processor, each frame of the received video data and one or more AR route position indicators in the navigation display, wherein the one or more AR route position indicators are aligned with each frame of the received video data in a rendered display based at least in part on the determined current fused camera pose.
2. The method of claim 1 , wherein the visual tracking of the features in the frames of the received video data comprises:
detecting features within a first frame and a second frame of the received video data;
detecting one or more bounding boxes of moving objects within the first frame and second frame of the received video data;
labeling the detected features within the first frame and second frame of the received video data based on and indicating a type for each detected feature;
filtering the labeled detected features based on the indicated type for each detected feature;
removing from the first frame and second frame of the received video data features inside of the one or more bounding boxes;
tracking changes in remaining features between the first frame and second frame of the received video data;
computing a camera essential matrix for the camera based on the tracked changes; and
calculating unit camera translation and rotation from the essential matrix.
3. The method of claim 1 , wherein determining the current fused camera pose for an initial frame of the received video data based on a combination of the received sensor data and the visual tracking of the features in the frames of the received video data further comprises:
detecting features within the initial frame;
calculating a position in the route being navigated by the vehicle based on the current geographic location indicated by the received sensor data;
applying a smoothing filter to the calculated position;
calculating a sensor camera pose from the filtered calculated position; and
using the calculated sensor camera pose as the determined current fused camera pose.
4. The method of claim 3 , wherein determining the current fused camera pose for a frame of the received video data after the initial frame further comprises:
calculating a position in the route being navigated by the vehicle based on the current geographic location indicated by the received sensor data;
applying a smoothing filter to the calculated position;
calculating a current sensor camera pose from the filtered calculated position;
finding positions of features based on the visual tracking of the features in the frames of the received video data;
determining whether visual tracking was successful;
in response to determining that visual tracking was successful, calculating unit camera translation and rotation based on the feature positions, scaling the calculated unit camera translation based on a speed of the vehicle indicated in the received sensor data, calculating an incremental visual camera pose from the scaled unit camera translation and rotation, calculating a current visual camera pose from the incremental visual camera pose and a previous camera pose, and combining the current sensor camera pose and the current visual camera pose into the determined current fused camera pose; and
in response to determining visual tracking was not successful, combining a most recent previous fused camera pose with the current sensor camera pose into the determined current fused camera pose.
5. The method of claim 4 , wherein the current sensor camera pose and current visual camera pose are weighted, wherein a weight assigned to the current visual camera is higher than a weight assigned to the current sensor camera pose and wherein combining the current sensor camera pose and the current visual camera pose into the determined current fused camera pose further comprises using the determined current fused camera pose as the previous camera pose for a subsequent frame of the received video data.
6. The method of claim 4 , wherein combining the current sensor camera pose and the current camera pose into the determined current fused camera pose comprises using the current visual camera pose as the previous camera pose for a subsequent frame of the received video data.
7. The method of claim 4 , wherein combining the current sensor camera pose and the current camera pose into the determined current fused camera pose comprises using both the current visual camera pose and the determined current fused camera pose as previous camera poses for a subsequent frame of the received video data.
8. A vehicle control system comprising:
a processor; and
a memory coupled with and readable by the processor and storing therein a set of instructions which, when executed by the processor, causes the processor to provide an Augmented Reality (AR) navigation display by:
initializing a camera of the vehicle and one or more sensors of a vehicle, wherein initializing the camera and the one or more sensors comprises calculating an intrinsic camera matrix for the camera, calculating route world positions for a route being navigated by the vehicle, initializing the camera for capturing video data, and initializing the one or more sensors for capturing sensor data;
receiving the sensor data from the one or more sensors of the vehicle, the received sensor data comprising an indication of a current geographic location of the vehicle;
receiving the video data from the camera of the vehicle, the received video data comprising a plurality of frames with each frame representing a scene from a route being navigated by the vehicle;
determining a current fused camera pose for each frame of the received video data based on a combination of the received sensor data and visual tracking of features in the frames of the received video data, wherein determining the current fused camera pose for each frame of the received video data based on a combination of the received sensor data and the received video data comprises performing the visual tracking of the features in the frames of the received video data, determining a visual camera pose for the camera for each frame of the received video data based on the visual tracking, determining a sensor camera pose for the camera for each frame of the received video data based on the received sensor data, and combining the determined visual camera pose and the determined sensor camera pose into the determined current fused camera pose for each frame of the received video data based on a previous camera pose; and
rendering each frame of the received video data and one or more AR route position indicators in the navigation display, wherein the one or more AR route position indicators are aligned with each frame of the received video data in a rendered display based at least in part on the determined current fused camera pose.
9. The vehicle of claim 8 , wherein the visual tracking of features in the frames of the received video data comprises:
detecting features within a first frame and a second frame of the received video data;
detecting one or more bounding boxes of moving objects within the first frame and second frame of the received video data;
labeling the detected features within the first frame and second frame of the received video data based on and indicating a type for each detected feature;
filtering the labeled features based on the indicated type for each detected feature;
removing from the first frame and second frame of the received video data features inside of the one or more bounding boxes;
tracking changes in remaining features between the first frame and second frame of the received video data;
computing a camera essential matrix for the camera based on the tracked changes; and
calculating unit camera translation and rotation from the essential matrix.
10. The vehicle of claim 8 , wherein determining the current fused camera pose for an initial frame of the received video data based on a combination of the received sensor data and the visual tracking of the features in the frames of the received video data further comprises:
detecting features within the initial frame;
calculating a position in the route being navigated by the vehicle based on the current geographic location indicated by the received sensor data;
applying a smoothing filter to the calculated position;
calculating a sensor camera pose from the filtered calculated position; and
using the calculated sensor camera pose as the determined current fused camera pose.
11. The vehicle of claim 10 , wherein determining the current fused camera pose for a frame of the received video data after the initial frame further comprises:
calculating a position in the route being navigated by the vehicle based on the current geographic location indicated by the received sensor data;
applying a smoothing filter to the calculated position;
calculating a current sensor camera pose from the filtered calculated position;
finding positions of features based on the visual tracking of the features in the frames of the received video data;
determining whether visual tracking was successful;
in response to determining that visual tracking was successful, calculating unit camera translation and rotation based on the feature positions, scaling the calculated unit camera translation based on a speed of the vehicle indicated in the received sensor data, calculating an incremental visual camera pose from the scaled unit camera translation and rotation, calculating a current visual camera pose from the incremental visual camera pose and a previous camera pose, and combining the current sensor camera pose and the current visual camera pose into the determined current fused camera pose;
in response to determining visual tracking was not successful, combining a most recent previous fused camera pose with the current sensor camera pose into the determined current fused camera pose; and
wherein the current sensor camera pose and current visual camera pose are weighted, wherein a weight assigned to the current visual camera is higher than a weight assigned to the current sensor camera pose and wherein combining the current sensor camera pose and the current visual camera pose into the determined current fused camera pose further comprises using one or both of the determined current fused camera pose or the current visual camera pose as the previous camera pose for a subsequent frame of the received video data.
12. A non-transitory computer-readable medium comprising a set of instructions stored therein which, when executed by a processor, causes the processor to provide an Augmented Reality (AR) navigation display by:
initializing a camera of a vehicle and one or more sensors of the vehicle, wherein initializing the camera and the one or more sensors comprises calculating an intrinsic camera matrix for the camera, calculating route world positions for a route being navigated by the vehicle, initializing the camera for capturing video data, and initializing the one or more sensors for capturing sensor data;
receiving the sensor data from the one or more sensors of the vehicle, the received sensor data comprising an indication of a current geographic location of the vehicle;
receiving the video data from the camera of the vehicle, the received video data comprising a plurality of frames with each frame representing a scene from a route being navigated by the vehicle;
determining a current fused camera pose for each frame of the received video data based on a combination of the received sensor data and visual tracking of features in the frames of the received video data, wherein determining the current fused camera pose for each frame of the received video data based on a combination of the received sensor data and the received video data comprises performing the visual tracking of the features in the frames of the received video data, determining a visual camera pose for the camera for each frame of the received video data based on the visual tracking, determining a sensor camera pose for the camera for each frame of the received video data based on the received sensor data, and combining the determined visual camera pose and the determined sensor camera pose into the determined current fused camera pose for each frame of the received video data based on a previous camera pose; and
rendering each frame of the received video data and one or more AR route position indicators in the navigation display, wherein the one or more AR route position indicators are aligned with each frame of the received video data in a rendered display based at least in part on the determined current fused camera pose.
13. The non-transitory computer-readable medium of claim 12 , wherein the visual tracking of the features in the frames of the received video data comprises:
detecting features within a first frame and a second frame of the received video data;
detecting one or more bounding boxes of moving objects within the first frame and second frame of the received video data;
labeling the detected features within the first frame and second frame of the received video data based on and indicating a type for each detected feature;
filtering the labeled features based on the indicated type for each detected feature;
removing from the first frame and second frame of the received video data features inside of the one or more bounding boxes;
tracking changes in remaining features between the first frame and second frame of the received video data;
computing a camera essential matrix for the camera based on the tracked changes; and
calculating unit camera translation and rotation from the essential matrix.
14. The non-transitory computer-readable medium of claim 12 , wherein determining the current fused camera pose for an initial frame of the received video data based on a combination of the received sensor data and the visual tracking of the features in the frames of the received video data further comprises:
detecting features within the initial frame;
calculating a position in the route being navigated by the vehicle based on the current geographic location indicated by the received sensor data;
applying a smoothing filter to the calculated position;
calculating a sensor camera pose from the filtered calculated position; and
using the calculated sensor camera pose as the determined current fused camera pose.
15. The non-transitory computer-readable medium of claim 14 , wherein determining the current fused camera pose for a frame of the received video data after the initial frame further comprises:
calculating a position in the route being navigated by the vehicle based on the current geographic location indicated by the received sensor data;
applying a smoothing filter to the calculated position;
calculating a current sensor camera pose from the filtered calculated position;
finding positions of features based on the visual tracking of the features in the frames of the received video data;
determining whether visual tracking was successful;
in response to determining that visual tracking was successful, calculating unit camera translation and rotation based on the feature positions, scaling the calculated unit camera translation based on a speed of the vehicle indicated in the received sensor data, calculating an incremental visual camera pose from the scaled unit camera translation and rotation, calculating a current visual camera pose from the incremental visual camera pose and a previous camera pose, and combining the current sensor camera pose and the current visual camera pose into the determined current fused camera pose; and
in response to determining visual tracking was not successful, combining a most recent previous fused camera pose with the current sensor camera pose into the determined current fused camera pose; and
wherein the current sensor camera pose and current visual camera pose are weighted, wherein a weight assigned to the current visual camera is higher than a weight assigned to the current sensor camera pose and wherein combining the current sensor camera pose and the current visual camera pose into the determined current fused camera pose further comprises using one or both of the determined current fused camera pose or the current visual camera pose as the previous camera pose for a subsequent frame of the received video data.