Methods and Systems for Generating a Map including Sparse and Dense Mapping Information

1 . A method performed by a device having a plurality of sensors, the method comprising:

receiving one or more outputs of the plurality of sensors at a first position of the device in an environment, wherein the one or more outputs comprise a first set of data corresponding to one or more visual features of the environment associated with the first position;

generating, based on correspondence in the one or more outputs of the plurality of sensors, a map of the environment comprising sparse mapping data indicative of the first set of data;

receiving one or more additional outputs of the plurality of sensors at a second position of the device in the environment, wherein the one or more additional outputs comprise a second set of data corresponding to one or more visual features of the environment associated with the second position;

modifying the map of the environment to further comprise sparse mapping data indicative of the second set of data;

receiving dense mapping information via one or more of the plurality of sensors, wherein the dense mapping information comprises data corresponding to objects in the environment in a manner representative of a relative structure of the objects in the environment; and

modifying the map of the environment to comprise the dense mapping information.

2 . The method of claim 1 , further comprising:

receiving additional data from a server, wherein the additional data comprises sparse mapping data and dense mapping information corresponding to a plurality of environments aggregated from a plurality of devices; and

based on the additional data, modifying the map of the environment to comprise the sparse mapping data and dense mapping information corresponding to the environment.

3 . The method of claim 1 , wherein the plurality of sensors include one or more of a gyroscope, accelerometer, camera, barometer, magnetometer, global positioning system (GPS), Wi-Fi sensor, near-field communication (NFC) sensor, and Bluetooth sensor.

4 . The method of claim 1 , further comprising:

determining an identification of a region in the environment based on one or more objects in the dense mapping information; and

providing, within the map of the environment, the identification of the region in the environment.

5 . The method of claim 1 , further comprising:

determining a relationship between the first position in the environment and the second position in the environment, wherein the relationship includes topological information corresponding to the environment; and

providing, within the sparse mapping, information associated with the relationship.

6 . The method of claim 1 , wherein receiving dense mapping information via one or more of the plurality of sensors comprises:

receiving the dense mapping information via a camera system, wherein the camera system is configured to capture depth images of the environment.

7 . The method of claim 6 , further comprising generating a three-dimensional geometry of at least a portion of the environment based on the depth images.

8 . The method of claim 1 , further comprising:

identifying, via one or more of the plurality of sensors, data in the sparse mapping corresponding to one or more moving objects in the environment; and

removing the data in the sparse mapping corresponding to the one or more moving objects.

9 . The method of claim 1 , further comprising:

determining, via one or more sensors of the plurality of sensors, a pose of the device, wherein the pose of the device comprises an orientation of the device relative to the one or more visual features in the environment and a direction of gravity;

identifying a geographic location of the one or more visual features in the sparse mapping; and

based at least in part on the pose of the device and the geographic location of the one or more visual features in the sparse mapping, determining a relative location of the device in the environment.

10 . The method of claim 1 , wherein receiving dense mapping information via one or more of the plurality of sensors comprises:

capturing the dense mapping information via at least one camera and a structured light sensor from one or more different angles relative to the environment and the device.

11 . The method of claim 1 , wherein modifying the map of the environment to further comprise sparse mapping data indicative of the second set of data further comprises:

determining a correspondence between one or more visual features of the environment associated with the second position to one or more visual features of the environment associated with the first position; and

based on the correspondence being above a threshold, modifying the map of the environment comprising the sparse mapping data to further comprise the second set of data.

12 . A system comprising:

a plurality of sensors;

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform functions comprising:

receiving one or more outputs of the plurality of sensors at a first position of the device in an environment, wherein the one or more outputs comprise a first set of data corresponding to one or more visual features of the environment associated with the first position;

generating, based on correspondence in the one or more outputs of the plurality of sensors, a map of the environment comprising sparse mapping data indicative of the first set of data;

receiving one or more additional outputs of the plurality of sensors at a second position of the device in the environment, wherein the one or more additional outputs comprise a second set of data corresponding to one or more visual features of the environment associated with the second position;

modifying the map of the environment to further comprise sparse mapping data indicative of the second set of data;

receiving dense mapping information via one or more of the plurality of sensors, wherein the dense mapping information comprises data corresponding to objects in the environment in a manner representative of a relative structure of the objects in the environment; and

modifying the map of the environment to comprise the dense mapping information.

13 . The system of claim 12 , wherein the functions further comprise:

receiving additional data from a server, wherein the additional data comprises sparse mapping data and dense mapping information corresponding to a plurality of environments aggregated from a plurality of devices; and

based on the additional data, modifying the map of the environment to comprise the sparse mapping data and dense mapping information corresponding to the environment.

14 . The system of claim 12 , wherein the functions further comprise:

determining, via one or more sensors of the plurality of sensors, a pose of the device, wherein the pose of the device comprises an orientation of the device relative to the one or more visual features in the environment and a direction of gravity;

identifying a geographic location of the one or more visual features in the sparse mapping; and

based at least in part on the pose of the device and the geographic location of the one or more visual features in the map of the environment, determining a relative location of the device in the environment.

15 . A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving one or more outputs of the plurality of sensors at a first position of the computing device in an environment, wherein the one or more outputs comprise a first set of data corresponding to one or more visual features of the environment associated with the first position;

generating, based on correspondence in the one or more outputs of the plurality of sensors, a map of the environment comprising sparse mapping data indicative of the first set of data;

receiving one or more additional outputs of the plurality of sensors at a second position of the computing device in the environment, wherein the one or more additional outputs comprise a second set of data corresponding to one or more visual features of the environment associated with the second position;

modifying the map of the environment to further comprise sparse mapping data indicative of the second set of data;

receiving dense mapping information via one or more of the plurality of sensors, wherein the dense mapping information comprises data corresponding to objects in the environment in a manner representative of relative structure of the objects in the environment; and

modifying the map of the environment to comprise the dense mapping information.

16 . The non-transitory computer readable medium of claim 15 , wherein the function of receiving dense mapping information via one or more of the plurality of sensors comprises:

receiving the dense mapping information via a camera system, wherein the camera system is configured to capture depth images of the environment.

17 . The non-transitory computer readable medium of claim 15 , wherein the functions further comprise: continuously updating the map of the environment based on additional outputs received from the plurality of sensors.

18 . The non-transitory computer readable medium of claim 15 , wherein the map of the environment further includes semantic mapping information, wherein the semantic mapping information identifies one or more regions in the map of the environment based on three dimensional geometry and one or more objects in the one or more regions in the environment.

19 . The non-transitory computer readable medium of claim 15 , wherein the function of receiving dense mapping information via one or more of the plurality of sensors comprises:

receiving data corresponding to one or more objects in the environment via one or more depth sensors; and

configuring the data corresponding to one or more objects with images captured by a camera to generate dense mapping information.

20 . The non-transitory computer readable medium of claim 15 , wherein the functions further comprise:

receiving data indicative of one or more Wi-Fi access points; and

modifying the map of the environment to further include the data indicative of the one or more Wi-Fi access points.