Systems and methods for simultaneous localization and mapping

1. A method comprising:

identifying, by one or more hardware processors, a first key image frame from a set of captured image frames captured by an image sensor; and

after detecting a movement of the image sensor from a first position in a physical environment to a second position in the physical environment:

identifying, by the one or more hardware processors, a second key image frame from the set of captured image frames;

performing, by the one or more hardware processors, feature matching on at least the first and second key image frames to identify a set of matching three-dimensional (3D) features in the physical environment;

generating, by the one or more hardware processors, a filtered set of matching 3D features by filtering out at least one erroneous feature, from the set of matching 3D features, based on a set of error criteria; and

determining, by the one or more hardware processors, a first set of six degrees of freedom (6DOF) parameters of the image sensor for the second key image frame and a set of 3D positions for the set of matching 3D features, the determining comprising performing a simultaneous localization and mapping (SLAM) process based on first inertial measurement unit (IMU) data captured by an inertial measurement unit (IMU), second IMU data captured by the IMU, and the filtered set of matching 3D features, the first IMU data being associated with the first key image frame, and the second IMU data being associated with the second key image frame.

2. The method of claim 1 , wherein the first IMU data comprises a first set of four degrees of freedom (4DOF) parameters of the image sensor, and the second IMU data comprises a second set of 4DOF parameters of the image sensor.

3. The method of claim 1 , wherein the image sensor and the IMU are included in a device.

4. The method of claim 3 , wherein the movement of the image sensor is caused by a human individual that is holding the device performing a sidestep.

5. The method of claim 4 , wherein the identifying the first key image frame comprises detecting a start impulse of the sidestep, and the first key image frame is a particular image frame, in the set of captured image frames, corresponding with the detected start impulse.

6. The method of claim 4 , wherein the identifying the second key image frame comprises detecting a finish impulse of the sidestep, and the second key image frame is a particular image frame, in the set of captured image frames, corresponding with the detected finish impulse.

7. The method of claim 1 , further comprising:

for each particular image frame added to the set of captured image frames:

determining, by the one or more hardware processors, whether a set of key image frame conditions is satisfied for the particular image frame;

in response to the set of key image frame conditions being satisfied for the particular image frame, identifying, by the one or more hardware processors, the particular image frame as a new key image frame and performing, by the one or more hardware processors, a full SLAM process cycle on the new key image frame; and

in response to the set of key image frame conditions not being satisfied for the particular image frame, performing, by the one or more hardware processors, a partial SLAM process cycle on the particular image frame as a non-key image frame; the partial SLAM process cycle comprising only a localization portion of a SLAM process.

8. The method of claim 7 , wherein the set of key image frame conditions comprises at least one of a new image frame meeting or exceeding a certain image quality, a minimum time has elapsed since the full SLAM process cycle was last performed, and a translation between a prior image frame and the new image frame meets or exceeds a minimum translation threshold.

9. The method of claim 7 , wherein performing the full SLAM process cycle on the new key image frame comprises:

identifying, by the one or more hardware processors, third IMU data, from the captured IMU data, associated with the new key image frame;

performing, by the one or more hardware processors, feature matching on the new key image frame and at least one prior image frame to identify a second set of matching 3D features in the physical environment;

determining, by the one or more hardware processors, a second set of 6DOF parameters of the image sensor for the new key image frame by performing the SLAM process on the new key image frame based on the second set of matching 3D features and the third IMU data;

generating, by the one or more hardware processors, a second filtered set of matching 3D features by filtering out at least one erroneous feature, from the second set of matching 3D features, based on a second set of error criteria and the second set of 6DOF parameters; and

determining, by the one or more hardware processors, a third set of 6DOF parameters of the image sensor for the new key image frame and a set of 3D positions of new 3D features in the physical environment by performing the SLAM process on all key image frames based on the second filtered set of matching 3D features and the third IMU data.

10. The method of claim 7 , wherein performing the partial SLAM process cycle on the non-key image frame comprises:

performing, by the one or more hardware processors; two-dimensional (2D) feature tracking on the non-key image frame, based on a set of 3D positions of new 3D features from the performing the full SLAM process cycle and a new key image frame most recently identified; to identify a set of 2D features;

determining, by the one or more hardware processors, a fourth set of 6DOF parameters of the image sensor for the non-key image frame by performing only the localization portion of the SLAM process based on the set of 21D features;

generating, by the one or more hardware processors, a filtered set of 2D features by filtering out at least one erroneous feature, from the set of 2D features, based on a third set of error criteria and the fourth set of 6DOF parameters; and

projecting, by the one or more hardware processors, a set of tracking points on the non-key image frame based on the filtered set of 21D features and the fourth set of 6DOF parameters.

11. A system comprising:

an image sensor;

a memory storing instructions; and

a hardware processor communicatively coupled to the memory and configured by the instructions to perform operations comprising:

for each particular image frame, in a set of captured image frames of a physical environment captured by the image sensor:

determining, by one or more hardware processors, whether a set of key image frame conditions is satisfied for the particular image frame;

in response to the set of key image frame conditions being satisfied for the particular image frame, identifying the particular image frame as a new key image frame and performing a full simultaneous localization and mapping (SLAM) process cycle on the new key image frame; and

in response to the set of key image frame conditions not being satisfied for the particular image frame, performing a partial SLAM process cycle on the particular image frame as a non-key image frame.

12. The system of claim 11 , further comprising:

detecting a movement of the image sensor, from a first pose in the physical environment to a second pose in the physical environment, based on inertial measurement unit (IMU) data captured from an inertial measurement unit (IMU) of the system, the IMU data comprising degrees of freedom (DOF) parameters of the image sensor; and

identifying a first key image frame and a second key image frame based on the movement, the first key image frame corresponding to a start impulse of the movement, and the second key image frame corresponding to a finish impulse of the movement.

13. The system of claim 11 , wherein the set of key image frame conditions comprises at least one of a new image frame meeting or exceeding a certain image quality, a minimum time has elapsed since the full SLAM process cycle was last performed, and a translation between a prior image frame and the new image frame meets or exceeds a minimum translation threshold.

14. The system of claim 11 , Wherein performing the full SLAM process cycle on the new key image frame comprises:

identifying second IMU data, from captured MU data, associated with the new key image frame, the captured IMU data being captured from an inertial measurement unit (IMU);

performing feature matching on the new key image frame and at least one prior image frame to identify a second set of matching 3D features in the physical environment;

determining a first set of six-degrees of freedom (6DOF) parameters of the image sensor for the new key image frame by performing a SLAM process on the new key image frame based on the second set of matching 3D features and the second IMU data;

generating a second filtered set of matching 3D features by filtering out at least one erroneous feature, from the second set of matching 3D features, based on a second set of error criteria and the first set of 6DOF parameters; and

determining a second set of 6DOF parameters of the image sensor for the new key image frame and a set of 3D positions of new 3D features in the physical environment by performing the SLAM process on all key image frames based on the second filtered set of matching 3D features and the second IMU data.

15. The system of claim 11 , wherein performing the partial SLAM process cycle on the non-key image frame comprises:

performing two-dimensional (2D) feature tracking on the non-key image frame, based on a set of 3D positions of new 3D features from the performing a full SLAM process cycle and a new key image frame most recently identified, to identify a set of 2D features;

determining a third set of six degrees of freedom (6DOF) parameters of the image sensor for the non-key image frame by performing only a localization portion of a SLAM process based on the set of 2D features;

generating a filtered set of 2D features by filtering out at least one erroneous feature, from the set of 2D features, based on a third set of error criteria and the third set of 6DOF parameters; and

projecting a set of tracking points on the non-key image frame based on the filtered set of 2D features and the third set of 6DOF parameters.

16. A system comprising:

an inertial measurement unit (IMU);

an image sensor;

a memory storing instructions; and

a hardware processor communicatively coupled to the memory and configured by the instructions to perform operations comprising:

identifying a first key image frame from a set of captured image frames captured by the image sensor; and

after detecting a movement of the image sensor from a first position, in a physical environment, to a second position in the physical environment:

identifying a second key image frame from the set of captured image frames;

performing feature matching on at least the first and second key image frames to identify a set of matching three-dimensional (3D) features in the physical environment;

generating a filtered set of matching 3D features by filtering out at least one erroneous feature, from the set of matching 3D features, based on a set of error criteria; and

determining a first set of six degrees of freedom (6DOF) parameters of the image sensor for the second key image frame and a set of 3D positions for the set of matching 3D features, the determining comprising performing a simultaneous localization and mapping (SLAM) process based on first IMU data captured by the IMU, second IMU data captured by the IMU, and the filtered set of matching 3D features, the first IMU data being associated with the first key image frame, and the second IMU data being associated with the second key image frame.

17. The system of claim 16 , wherein the first IMU data comprises a first set of four degrees of freedom (4DOF) parameters of the image sensor, and the second IMU data comprises a second set of 41DOF parameters of the image sensor.

18. The system of claim 16 , wherein the movement of the image sensor is caused by a human individual that is holding the system performing a sidestep.

19. The system of claim 18 , wherein the identifying the first key image frame comprises detecting a start impulse of the sidestep, and the first key image frame is a particular image frame, in the set of captured image frames, corresponding with the detected start impulse.

20. The system of claim 18 , wherein the identifying the second key image frame comprises detecting a finish impulse of the sidestep, and the second key image frame is a particular image frame, in the set of captured image frames, corresponding with the detected finish impulse.