Cloud assisted generation of local map data using novel viewpoints
An augmented reality system generates computer-mediated reality on a client device. The client device has sensors including a camera configured to capture image data of an environment and a location sensor to capture location data describing a geolocation of the client device. The client device creates a three-dimensional (3-D) map with the image data and the location data for use in generating virtual objects to augment reality. The client device transmits the created 3-D map to an external server that may utilize the 3-D map to update a world map stored on the external server. The external server sends a local portion of the world map to the client device. The client device determines a distance between the client device and a mapping point to generate a computer-mediated reality image at the mapping point to be displayed on the client device.
1. A non-transitory computer-readable storage medium storing instructions for generating map data on a client device, the instructions, when executed by the client device, cause the client device to perform operations comprising:
receiving an image depicting a view of an environment;
identifying the view of the environment as a novel viewpoint based on (a) a comparison of the image depicting the view of the environment to a preexisting image of the environment and (b) a determination that the view of the environment has not been received by the client device for at least a threshold amount of time;
after identifying the view of the environment as the novel viewpoint, generating novel local map data from the image depicting the view of the environment; and
providing the novel local map data to a server for fusing with other map data.
2. The non-transitory computer-readable storage medium of claim 1 , wherein the other map data is generated from data corresponding to at least the preexisting image of the environment.
3. The non-transitory computer-readable storage medium of claim 1 , wherein the operations further comprise:
receiving a portion of the other map data from the server;
determining a distance between a mapping point in the portion of the other map data and a spatial position of the client device in the portion of the other map data;
generating a computer mediated reality image at the mapping point in the portion of the other map data based at least in part on the distance between the mapping point and the spatial position of the client device; and
displaying the computer mediated reality image at the mapping point.
4. The non-transitory computer-readable storage medium of claim 3 , wherein the operations further comprise:
determining a subsequent distance between the mapping point in the portion of the other map data and a subsequent spatial position of the client device in the portion of the other map data;
adjusting the computer mediated reality image at the mapping point in the portion of the other map data based on subsequent location data; and
displaying the adjusted computer mediated reality image at the mapping point.
5. The non-transitory computer-readable storage medium of claim 3 , wherein the operations further comprise transmitting the image depicting the view of the environment to the server, wherein the portion of the other map data is selected based on the image depicting the view of the environment.
6. The non-transitory computer-readable storage medium of claim 1 , wherein identifying the view of the environment as the novel viewpoint based on the comparison of the image depicting the view of the environment to the preexisting image of the environment comprises:
determining an overlap between fields of view of the image depicting the view of the environment and the preexisting image is below an overlap threshold.
7. The non-transitory computer-readable storage medium of claim 1 , wherein identifying the view of the environment as the novel viewpoint comprises:
retrieving the preexisting image and one or more other preexisting images, the retrieved preexisting images associated with geographic positions within a threshold radius of a geographic position of the client device;
filtering the retrieved preexisting images based on a comparison of a set of features identified in the image depicting the view of the environment to features identified in the retrieved preexisting images; and
determining that the image depicting the view of the environment does not match the filtered preexisting images.
8. The non-transitory computer-readable storage medium of claim 1 , wherein generating the novel local map data comprises:
identifying one or more objects in the environment from the image depicting the view of the environment;
determining spatial positions for the one or more objects from the image depicting the view of the environment; and
generating a 3D point cloud comprising a set of 3-D points for each of the one or more objects.
9. The non-transitory computer-readable storage medium of claim 8 , wherein generating the novel local map data further comprises:
classifying each object of the one or more objects into one of a plurality of object types, the plurality of object types including a stationary type describing objects that are expected to remain in substantially a same spatial position.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the plurality of object types further includes a temporary type describing objects that are not expected to remain in substantially the same spatial position.
11. The non-transitory computer-readable storage medium of claim 1 , wherein the other map data is subdivided into layers of tiles, the layers correlating to different zoom levels.
12. The non-transitory computer-readable storage medium of claim 1 , wherein generating the novel local map data from the image depicting the view of the environment comprises generating a three-dimensional (3D) point cloud based on the image depicting the view of the environment.
13. A method of generating map data, the method comprising:
receiving, by a client device, an image depicting a view of an environment;
identifying the view of the environment as a novel viewpoint based on (a) a comparison of the image depicting the view of the environment to a preexisting image of the environment and (b) a determination that the view of the environment has not been received by the client device for at least a threshold amount of time;
after identifying the view of the environment as the novel viewpoint, generating novel local map data from the image depicting the view of the environment; and
providing the novel local map data to a server for fusing with other map data.
14. The method of claim 13 , further comprising:
receiving a portion of the other map data from the server;
determining a distance between a mapping point in the portion of the other map data and a spatial position of the client device in the portion of the other map data;
generating a computer mediated reality image at the mapping point in the portion of the other map data based at least in part on the distance between the mapping point and the spatial position of the client device; and
displaying the computer mediated reality image at the mapping point.
15. The method of claim 14 , further comprising:
determining a subsequent distance between the mapping point in the portion of the other map data and a subsequent spatial position of the client device in the portion of the other map data;
adjusting the computer mediated reality image at the mapping point in the portion of the other map data based on subsequent location data; and
displaying the adjusted computer mediated reality image at the mapping point.