Cloud enabled augmented reality

1. A method of generating computer mediated reality data on a client device, the method comprising:

capturing image data with a camera integrated in the client device, the image data representing a near real-time view of an environment around the client device;

capturing location data with a location sensor integrated in the client device, the location data describing a spatial position of the client device in the environment;

generating local map data based on the image data and the location data, the local map data including one or more three-dimensional (3D) point clouds spatially describing one or more objects in the environment around the client device;

accessing a set of viewpoints stored locally on the client device based on the spatial position of the client device;

identifying a first image in the image data as a novel viewpoint based at least in part on the first image having less than a minimum overlap in field of view with each viewpoint in the set of viewpoints;

transmitting the local map data and the first image as the novel viewpoint to an external server;

receiving a local portion of world map data at the client device from the external server, wherein the world map data is generated by fusing a plurality of local maps generated by one or more client devices and viewpoints captured by one or more client devices, wherein the local portion is selected based on the transmitted local map data;

determining a distance between a mapping point in the local portion of world map data and the spatial position of the client device in the local portion of world map data based on location data;

generating a computer mediated reality image at the mapping point in the local portion of world map data based on the image data, the location data, and the distance between the mapping point and the spatial position of the client device; and

displaying the computer mediated reality image at the mapping point.

2. The method of claim 1 , further comprising transmitting the image data to the external server, wherein the local portion of world map data is selected further based on the image data.

3. The method of claim 1 , further comprising:

identifying a second image in the image data as a second novel viewpoint based at least in part on the second image having less than the minimum overlap in field of view with the first set of viewpoints and the first image; and

transmitting the second image as the second novel viewpoint to the external server.

4. The method of claim 1 , wherein the computer mediated reality image comprises a virtual object that is fixed at the mapping point in the local portion of world map data.

5. The method of claim 4 , further comprising receiving a virtual object from the external server based on the image data and the location data, wherein the computer mediated reality image comprises the received virtual object.

6. The method of claim 1 , wherein generating local map data based on the image data and the location data further comprises:

identifying one or more objects in the environment based on the image data;

determining one or more spatial positions for each of the objects based on the image data and the location data; and

generating a 3D point cloud comprising a set of 3-D points for each of the objects.

7. The method of claim 6 , wherein generating local map data based on the image data and the location data further comprises:

classifying each object into one of a plurality of object types, the plurality of objects types including a stationary type describing objects that are expected to remain in substantially a same spatial position.

8. The method of claim 1 , further comprising:

capturing subsequent location data with the location sensor describing a subsequent spatial position of the client device in the environment;

determining a subsequent distance between the mapping point in the local portion of world map data and the subsequent spatial position of the client device based on location data and the local portion of world map data;

adjusting the computer mediated reality image at the mapping point in the local portion of world map data based on the subsequent location data; and

displaying the adjusted computer mediated reality image at the mapping point.

9. A method comprising:

storing world map data that describes an aggregate of a plurality of environments at a plurality of geolocations and viewpoint data describing images captured at the plurality of environments;

receiving, from a client device, local map data, image data of an environment captured by a camera on the client device, location data describing a spatial position of the client device captured with a location sensor, and one or more novel viewpoints, wherein the local map data is generated by the client device based on the image data and the location data and includes one or more 3D point clouds spatially describing one or more objects in the environment around the client device;

identifying a set of viewpoints from the viewpoint data based on the location data;

for each novel viewpoint, determining whether the novel viewpoint has less than a minimum overlap in field of view with each viewpoint in the set of viewpoints;

responsive to determining that a first novel viewpoint has less than the minimum overlap in field of view with each viewpoint in the set of viewpoints, adding the first novel viewpoint to the set of viewpoints;

retrieving a local portion of world map data based on the location data and the local map data;

updating the local portion of the world map data with the local map data by adding one or more of the 3D point clouds in the local map data to the world map data; and

transmitting the local portion of the world map data to the client device.

10. The method of claim 9 , further comprising:

generating a virtual object to be displayed by the client device in the environment based on the image data and the location data; and

transmitting the virtual object to the client device.

11. The method of claim 9 , further comprising:

determining, for each novel viewpoint, whether a subset of the set of viewpoints having more than the minimum overlap in field of view of the novel viewpoint was captured over a threshold interval of time the novel viewpoint; and

responsive to determining that a first subset of the set of viewpoints having more than the minimum overlap in field of view of a second novel viewpoint was captured over the threshold interval of time from the second novel viewpoint, adding the second novel viewpoint to the set of viewpoints.

12. A non-transitory computer-readable storage medium storing instructions for generating computer mediated reality data on a client device, the instructions, that when executed by a processor, cause the processor to perform operations comprising:

capturing image data with a camera integrated in the client device, the image data representing a near real-time view of an environment around the client device;

capturing location data with a location sensor integrated in the client device, the location data describing a spatial position of the client device in the environment;

generating local map data based on the image data and the location data, the local map data including one or more three-dimensional (3D) point clouds spatially describing one or more objects in the environment around the client device;

accessing a set of viewpoints stored locally on the client device based on the spatial position of the client device;

identifying a first image in the image data as a novel viewpoint based at least in part on the first image having less than a minimum overlap in field of view with each viewpoint in the set of viewpoints;

transmitting the local map data and the first image as the novel viewpoint to an external server;

receiving a local portion of world map data at the client device from the external server, wherein the world map data is generated by fusing a plurality of local maps generated by one or more client devices and viewpoints captured by one or more client devices, wherein the local portion is selected based on the transmitted local map data;

determining a distance between a mapping point in the local portion of world map data and the spatial position of the client device in the local portion of world map data based on location data;

generating a computer mediated reality image at the mapping point in the local portion of world map data based on the image data, the location data, and the distance between the mapping point and the spatial position of the client device; and

displaying the computer mediated reality image at the mapping point.

13. The storage medium of claim 12 , further comprising transmitting the image data to the external server, wherein the local portion of world map data is selected further based on the image data.

14. The storage medium of claim 12 , further comprising:

identifying a second image in the image data as a second novel viewpoint based at least in part on the second image having less than the minimum overlap in field of view with the first set of viewpoints and the first image; and

transmitting the second image as the second novel viewpoint to the external server.

15. The storage medium of claim 12 , wherein the computer mediated reality image comprises a virtual object that is fixed at the mapping point in the local portion of world map data.

16. The storage medium of claim 15 , further comprising receiving a virtual object from the external server based on the image data and the location data, wherein the computer mediated reality image comprises the received virtual object.

17. The storage medium of claim 12 , wherein generating local map data based on the image data and the location data further comprises:

identifying one or more objects in the environment based on the image data;

determining one or more spatial positions for each of the objects based on the image data and the location data; and

generating a 3D point cloud comprising a set of 3-D points for each of the objects.

18. The storage medium of claim 17 , wherein generating local map data based on the image data and the location data further comprises:

classifying each object into one of a plurality of object types, the plurality of objects types including a stationary type describing objects that are expected to remain in substantially a same spatial position.

19. The storage medium of claim 12 , further comprising:

capturing subsequent location data with the location sensor describing a subsequent spatial position of the client device in the environment;

determining a subsequent distance between the mapping point in the local portion of world map data and the subsequent spatial position of the client device based on location data and the local portion of world map data;

adjusting the computer mediated reality image at the mapping point in the local portion of world map data based on the subsequent location data; and

displaying the adjusted computer mediated reality image at the mapping point.