Caching and updating of dense 3D reconstruction data
A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications. Between sessions of an XR device, blocks may be persisted on the device or in remote storage in one or more cache layers. The device may, upon starting a new session, selectively use the blocks from one or more layers of the cache.
1. A portable electronic system comprising:
a sensor configured to capture three-dimensional (3D) information about objects in a physical world;
an active memory;
a local memory;
a transceiver configured for communication over a computer network with remote memory; and
a processor communicatively coupled to the sensor, the active memory, the local memory, and the transceiver, and configured to execute computer executable instructions to provide a 3D representation of a portion of the physical world based at least in part on the 3D information about the objects in the physical world, the portion of the physical world being within a field of view of a user of the portable electronic system, wherein:
the 3D representation of the portion of the physical world comprises a plurality of blocks, each block having values representing objects in a region of the portion of the physical world at a point in time and comprising a volumetric representation of the region of the portion of the physical world at the point in time; and
the computer executable instructions comprise instructions for:
persisting blocks of the plurality of blocks in at least one of the active memory and the local memory when the portable electronic system changes to an inactive operation status.
2. The portable electronic system of claim 1 , wherein the inactive operation status is shutdown or sleep and persisting the blocks comprises copying the blocks to a non-volatile memory.
3. The portable electronic system of claim 2 , wherein persisting the plurality of blocks in at least one of the active memory, the local memory and the remote memory comprises,
when the inactive operation status is sleep, persisting the plurality of blocks in respective memories in which the plurality of blocks currently exist such that the plurality of blocks are accessible by the portable electronic system upon waking of the portable electronic system.
4. The portable electronic system of claim 2 , wherein persisting the plurality of blocks in at least one of the active memory, the local memory and the remote memory comprises,
when the inactive operation status is shutdown, persisting at least a portion of blocks in the local memory by copying the blocks to the remote memory, and persisting blocks in the active memory by copying the blocks to the local memory and/or the remote memory.
5. The portable electronic system of claim 1 , wherein the computer executable instructions comprise instructions for:
computing one or more new blocks based at least in part on incoming 3D information about objects in the physical world, and
selecting, from the persisted blocks, blocks at least in part overlapping with the one or more new blocks such that persisted blocks around the portable electronic system can be retrieved.
6. The portable electronic system of claim 5 , wherein selecting, from the persisted blocks, the blocks at least in part overlapping with the one or more new blocks comprises comparing the one or more new blocks with the persisted blocks.
7. The portable electronic system of claim 5 , wherein the computer executable instructions comprise instructions for:
updating the plurality of blocks with the one or more new blocks.
8. The portable electronic system of claim 6 , wherein comparing the one or more new blocks with the persisted blocks comprises
comparing the one or more new blocks to blocks persisted in the active memory.
9. The portable electronic system of claim 8 , wherein comparing the one or more new blocks with the persisted blocks comprises
when the active memory has no block at least in part overlapping with the one or more new blocks, comparing the one or more new blocks to blocks persisted in the local memory.
10. The portable electronic system of claim 9 , wherein comparing the one or more new blocks with the persisted blocks comprises
when the local memory has no block at least in part overlapping with the one or more new blocks, comparing the one or more new blocks to blocks persisted in the remote memory.
11. A method of operating a portable electronic system in an augmented reality system, the method comprising:
with at least one processor:
obtaining a plurality of blocks comprising a three-dimensional (3D) representation of surfaces in a physical world, each block comprising a volumetric representation of a portion of the physical world;
storing the plurality of blocks in an active memory;
using blocks from the active memory for rendering virtual content to a user of the portable electronic system;
upon a transition to an inactive operation status at a first time, selectively persisting the blocks in the active memory; and
upon a transition to an active operation status at a second time, after the first time, determining whether to use the persisted blocks for rendering virtual content to a user of the portable electronic system.
12. The method of claim 11 , wherein the blocks are mesh blocks.
13. The method of claim 11 , wherein the plurality of blocks correspond to spaces of a same volume.
14. The method of claim 11 , further comprising:
capturing 3D information with a sensor of the portable electronic system;
replacing a subset of the plurality of blocks in active memory with modified blocks computed based on the captured 3D information;
and
paging blocks between the active memory and a local memory based on current or projected field of view of the user of the portable electronic system.
15. The method of claim 11 , wherein using blocks from the active memory for rendering virtual content to the user comprises:
using the blocks for visual occlusion processing, computing physics-based interactions and/or environmental reasoning.
16. A non-transitory computer-readable medium with instructions stored thereon, that when executed on a processor, perform the acts comprising:
obtaining a plurality of blocks comprising a three-dimensional (3D) representation of surfaces in a physical world, each block comprising a volumetric representation of a portion of the physical world at a point in time;
storing the plurality of blocks in an active memory;
using blocks from the active memory for rendering virtual content to a user of the portable electronic system;
upon a transition to an inactive operation status at a first time, selectively transferring the blocks in the active memory to a non-volatile memory; and
upon a transition to an active operation status at a second time, after the first time, determining whether to use the transferred blocks for rendering virtual content to a user of the portable electronic system.
17. The non-transitory computer-readable medium of claim 16 , wherein the blocks are mesh blocks.
18. The non-transitory computer-readable medium of claim 16 , wherein
each of the plurality of blocks in the subset:
represents a respective portion of the physical world at the first time; and
is replaced by a modified block representing the respective portion of the physical world at the second time.
19. The non-transitory computer-readable medium of claim 16 , wherein the acts comprise:
replacing a subset of the plurality of blocks in the active memory with modified blocks computed based on 3D information captured with a sensor.
20. The non-transitory computer-readable medium of claim 16 , wherein using blocks from the active memory for rendering virtual content to the user comprises:
using the blocks for visual occlusion processing, and
computing physics-based interactions and/or environmental reasoning.