Multi-camera high speed simultaneous localization and mapping (SLAM) for a head mounted display
In one or more embodiments, instructions that, when executed by a processor, cause the processor to: locate, based on a sensor fusion of a second camera and a second sensor, a first compute device in a map of a 3D scene to define a first device location; calculate, based on the map of the 3D scene and a first sensor, a relative pose of a second compute device with respect to a first compute device location; determine, based on the relative pose, a region of overlap between a FOV of the first camera and a FOV of the second camera; identify, based on the region of overlap, an occluded portion of the second FOV; and send a signal to cause the display to project a plurality of image frames within the second FOV and to reproject the visible portion of the first FOV.
1 . A method, comprising:
generating a map of a 3D scene based on a first plurality of image frames of a first sensor of a first compute device;
locating the first compute device in the map of the 3D scene to define a first compute device location, based on a sensor fusion of the first sensor of the first compute device and a second sensor of the first compute device;
matching a plurality of features in a second plurality of image frames of a sensor of a second compute device to a plurality of features in the map of the 3D scene, to define a region of overlap between the first plurality of image frames and the second plurality of image frames;
calculating, based on the region of overlap, a relative pose of the second compute device with respect to the first compute device location;
identifying a portion of the 3D scene in the first plurality of image frames that is representative of a field of view of the sensor of the second compute device, based on the relative pose of the second compute device;
sending a first signal to display the first plurality of image frames; and
sending a second signal to display a visual indicator that indicates the portion of the 3D scene in the first plurality of image frames.
2 . The method of claim 1 , wherein the second sensor is an inertial measurement unit (IMU).
3 . The method of claim 1 , wherein the first compute device is a head mounted display (HMD), and the second compute device is (1) communicatively coupled to the HMD and (2) physically uncoupled from the HMD.
4 . The method of claim 1 , wherein the first sensor of the first compute device is a first infrared (IR) camera and the sensor of the second compute device is a second IR camera.
5 . The method of claim 1 , wherein the first signal and the second signal are sent in substantially real-time relative to the first plurality of image frames and the second plurality of image frames.
6 . The method of claim 1 , further comprising:
updating the map of the 3D scene with inert elements and non-inert elements based on an image segmentation of at least one of the first plurality of image frames and the second plurality of image frames.
7 . An apparatus, comprising:
a second compute device including:
a first camera having a first Field-of-View (FOV), and
a first sensor; and
a first compute device configured to be communicatively coupled to the second compute device, the first compute device including:
a second camera having a second FOV greater than the first FOV,
a second sensor,
a display,
a processor, and
a non-transitory, processor-readable medium storing instructions that, when executed by the processor, cause the processor to:
receive a map of a 3D scene;
locate, based on a sensor fusion of the second camera and the second sensor, the first compute device in the map of the 3D scene to define a first device location;
calculate, based on the map of the 3D scene and the first sensor, a relative pose of the second compute device with respect to the first compute device location;
determine, based on the relative pose of the second compute device, a region of overlap between the first FOV and the second FOV;
identify, based on the region of overlap, an occluded portion of the second FOV, the occluded portion of the second FOV being a visible portion of the first FOV; and
send a signal to cause the display to project a plurality of image frames within the second FOV and to reproject the visible portion of the first FOV.
8 . The apparatus of claim 7 , wherein the first sensor is a first inertial measurement unit (IMU) and the second sensor is a second IMU.
9 . The apparatus of claim 7 , wherein the first compute device is a head mounted display (HMD) and the second compute device is physically uncoupled from the HMD.
10 . The apparatus of claim 7 , wherein the processor sends a signal to cause the display to reproject the visible portion of the first FOV in substantially real-time.
11 . The apparatus of claim 7 , wherein the first camera is a first infrared (IR) camera and the second camera is a second IR camera.
12 . The apparatus of claim 7 , wherein the non-transitory, processor-readable medium stores further instructions that cause the processor further to:
send a signal to cause the display to project a bounding box, the bounding box being associated with the visible portion of the first FOV.
13 . The apparatus of claim 7 , wherein the non-transitory, processor-readable medium stores further instructions that cause the processor further to:
update the map of the 3D scene with inert elements and non-inert elements based on an image segmentation of at least one of the first plurality of image frames and the second plurality of image frames.
14 . A non-transitory, processor-readable medium storing instructions that, when executed by a processor, cause the processor to:
receive a first plurality of image frames from a first sensor of a first compute device;
receive a second plurality of image frames from a second sensor of a second compute device;
determine a relative image quality between the first plurality of image frames and the second plurality of image frames to define a quality plurality of image frames;
generate a map of a 3D scene based on the quality plurality of image frames;
locate, based on a sensor fusion of the first sensor and a third sensor of the first compute device, the first compute device in the map of the 3D scene to define a first compute device location;
match a plurality of features in the second plurality of image frames to a plurality of features in the map of the 3D scene, to define a region of overlap between the first plurality of image frames and the second plurality of image frames;
calculate, based on the region of overlap, a relative pose of the second compute device with respect to the first compute device location;
send a first signal to cause a display to project the first plurality of image frames; and
send a second signal to cause the display to project a visual indicator of a field of view (FOV) of the second compute device, based on the relative pose of the second compute device.
15 . The non-transitory, processor-readable medium of claim 14 , wherein the third sensor is an inertial measurement unit (IMU).
16 . The non-transitory, processor-readable medium of claim 14 , wherein the first compute device is a head mounted display (HMD) operably coupled to the processor, the display is a display of the HMD, and the second compute device is (1) physically uncoupled from the HMD and (2) communicatively coupled to the HMD.
17 . The non-transitory, processor-readable medium of claim 14 , wherein the first signal and the second signal are sent in substantially real-time relative to the first plurality of image frames and the second plurality of image frames.
18 . The non-transitory, processor-readable medium of claim 14 , wherein the first sensor is a first infrared (IR) camera and the second sensor is a second IR camera.
19 . The non-transitory, processor-readable medium of claim 14 , further storing instructions to cause the processor to:
send a third signal to cause the display to project a bounding box associated with the region of overlap, based on the relative pose of the second compute device.
20 . The non-transitory, processor-readable medium of claim 14 , further storing instructions to cause the processor to:
send a signal to cause the display to project the second plurality of image frames of the second sensor of the second compute device in response to a failure of the first sensor of the first compute device.