Caching and updating of dense 3D reconstruction data
A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.
1. A method for providing a 3D representation of a physical world, the method comprising:
a sensor capturing 3D information about objects in a portion of the physical world that is within a first field-of-view (FOV) of a first user;
a processor representing the portion of the physical world as a first plurality of blocks of 3D reconstruction data from the first FOV, each block of 3D reconstruction data corresponding to a predetermined volume of the portion of the physical world; and
the processor persisting at least one of the first plurality of blocks of 3D reconstruction data from the first FOV in a cache when the user has a second FOV, the second FOV being different from the first FOV,
wherein the persisted at least one of the first plurality of blocks of 3D reconstruction data represents the differences between the first FOV and the second FOV.
2. The method of claim 1 , wherein the first plurality of blocks of 3D reconstruction data are in the format of meshes, pointe clouds, or voxels.
3. The method of claim 1 , wherein the cache is a local cache of a portable electronic device on the first user or a remote cache of a computer network accessible by the portable electronic device on the first user.
4. The method of claim 3 , wherein the remote cache of the computer network persists at least one block of 3D reconstruction data from a second user.
5. The method of claim 4 , wherein the persisted at least one of the first plurality of blocks of 3D reconstruction data is merged with the at least one block of 3D reconstruction data from the second user based on a same coordinate system such that both the first user and the second user can request the merged blocks of 3D reconstruction from the remote cache of the computer network.
6. The method of claim 1 , further comprising the processor implementing a service that provides the 3D representation of the portion of the physical world to an application.
7. The method of claim 6 , further comprising the processor executing the application.
8. The method of claim 1 , further comprising the processor providing information through an application programming interface (API) executing on the processor.
9. The method of claim 1 , further comprising the processor creating a coordinate frame in the portion of the physical world,
wherein each of the first plurality of blocks represents objects in different regions of the portion of the physical world that is identifiable using the coordinate frame.
10. The method of claim 1 , further comprising:
the processor processing the 3D information into voxels, bricks, and tiles, each voxel comprising one or more signed distance functions (SDFs), each brick comprising one or more voxels, and each tile comprising one or more bricks, and
the processor generating each block of the plurality of blocks by converting one or more tiles into meshes,
wherein the 3D information is paged out of a local memory on the basis of tiles and/or blocks.