Local physical environment modeling in extended reality environments
Disclosed herein is an environmental scanning tool that generates a digital model representing the surroundings of a user of an extended reality head-mounted display device. The environment is imaged in both a depth map and in visible light for some select objects of interest. The selected objects exist within the digital model at higher fidelity and resolution than the remaining portions of the model in order to manage the storage size of the digital model. In some cases, the objects of interest are selected, or their higher fidelity scans are directed, by a remote user. The digital model further includes time stamped updates of the environment such that users can view a state of the environment according to various timestamps.
1. A method comprising:
providing, via a head mounted extended reality device (“XR device”), directions for a user to aim one or more external facing environment cameras of the XR device at a physical object within a local physical environment of the user, wherein multiple images captured by one or more of the cameras are each a depth map frame;
generating, by combining one or more of the depth map frames and a corresponding visible light frame, a digital representation of the physical object within the local physical environment of the user, wherein the visible light frame being combined with its corresponding depth map frame provides color to a portion of the digital representation of the physical object; and
transmitting the digital representation of the physical object to an other device.
2. The method of claim 1 , wherein the other device is a second XR device worn by a second user and the digital representation of the physical object is displayed, via the second XR device, as a 3D model.
3. The method of claim 1 ,
wherein the other device is a second XR device worn by a second user; and
wherein the method further comprises causing a system to:
identify a make and model of the physical object using an image recognition system; and
cause a textual display of the make and model of the physical object to be rendered on a display of the second XR device worn by the second user, wherein the textual display is positioned to appear adjacent to the digital representation of the physical object.
4. The method of claim 1 , further comprising identifying a make and model of the physical object using image recognition performed on the digital representation of the physical object.
5. The method of claim 1 ,
wherein the other device is a second XR device worn by the second user; and
wherein the method further comprises causing a system to:
identify a make and model of the physical object;
render a textual display of the make and model of the physical object to a display of the second XR device worn by the second user; and
update a user account of the first user confirming a status of the physical object in response to input by the second user.
6. The method of claim 1 , further comprising causing a system to:
identify one or more characteristics of the physical object using image recognition;
cause a textual display of the one or more characteristics of the physical object to be rendered on a display of the other device; and
generate a photograph of the physical object based on the digital representation of the physical object.
7. The method of claim 1 , further comprising:
subsequent to the transmitting the digital representation, and in response to a request from an entity, capturing an updated local physical environment of the user via the one or more external facing environment cameras of the XR device;
generating, using the captured updated local physical environment, a digital representation of an updated version of the physical object; and
transmitting the digital representation of the updated version of the physical object;
wherein, in response to receiving the digital representation of the updated version of the physical object, an insurance claim is established or modified based on a difference between the digital representation of the physical object and the digital representation of the updated version of the physical object.
8. The method of claim 1 , further comprising:
causing identification of characteristics of the physical object via computer vision executed with aspects of the local physical environment; and
receiving, by the XR device, hand gesture input via the one or more external facing environment cameras, wherein the hand gesture input confirms the identification of the characteristics of the physical object.
9. The method of claim 1 , further comprising:
causing identification of a make and model of the physical object via computer vision; and
causing display of a monetary value for the physical object, the monetary value determined based on the make and model.
10. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process comprising:
providing, via a head mounted extended reality device (“XR device”), directions for a user to aim one or more external facing environment cameras of the XR device at a physical object within a local physical environment of the user, wherein multiple images captured by one or more of the cameras are each a depth map frame;
generating, by combining one or more of the depth map frames and a corresponding visible light frame, a digital representation of the physical object within the local physical environment of the user, wherein the visible light frame being combined with its corresponding depth map frame provides color to a portion of the digital representation of the physical object; and
transmitting the digital representation of the physical object to an other device.
11. The computer-readable storage medium of claim 10 , wherein the process further comprises:
causing identification of a make and model of the physical object via computer vision; and
causing display of a monetary value for the physical object, the monetary value determined based on the make and model.
12. The computer-readable storage medium of claim 10 , wherein the other device is a second XR device worn by a second user and the digital representation of the physical object is displayed, via the second XR device, as a 3D model.
13. The computer-readable storage medium of claim 10 ,
wherein the other device is a second XR device worn by a second user; and
wherein the process further comprises causing a system to:
identify a make and model of the physical object using an image recognition system; and
cause a textual display of the make and model of the physical object to be rendered on a display of the second XR device worn by the second user, wherein the textual display is positioned adjacent to the digital representation of the physical object.
14. The computer-readable storage medium of claim 10 , wherein the process further comprises identifying a make and model of the physical object using image recognition performed on the digital representation of the physical object.
15. The computer-readable storage medium of claim 10 ,
wherein the other device is a second XR device worn by the second user; and
wherein the process further comprises causing a system to:
identify a make and model of the physical object;
render a textual display of the make and model of the physical object to a display of the second XR device worn by the second user; and
update a user account of the first user confirming a status of the physical object in response to input by the second user.
16. The computer-readable storage medium of claim 10 , wherein the process further comprises causing a system to:
identify one or more characteristics of the physical object using image recognition;
cause a textual display of the one or more characteristics of the physical object to be rendered on a display of the other device; and
generate a photograph of the physical object based on the digital representation of the physical object.
17. The computer-readable storage medium of claim 10 , wherein the process further comprises:
subsequent to the transmitting the digital representation, and in response to a request from an entity, capturing an updated local physical environment of the user via the one or more external facing environment cameras of the XR device;
generating, using the captured updated local physical environment, a digital representation of an updated version of the physical object; and
transmitting the digital representation of the updated version of the physical object;
wherein, in response to receiving the digital representation of the updated version of the physical object, an insurance claim is established or modified based on a difference between the digital representation of the physical object and the digital representation of the updated version of the physical object.
18. The computer-readable storage medium of claim 10 , wherein the process further comprises:
causing identification of characteristics of the physical object via computer vision executed with aspects of the local physical environment; and
receiving, by the XR device, hand gesture input via the one or more external facing environment cameras, wherein the hand gesture input confirms the identification of the characteristics of the physical object.
19. A computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
providing, via a head mounted extended reality device (“XR device”), directions for a user to aim one or more external facing environment cameras of the XR device at a physical object within a local physical environment of the user, wherein multiple images captured by one or more of the cameras are each a depth map frame;
generating, by combining one or more of the depth map frames and a corresponding visible light frame, a digital representation of the physical object within the local physical environment of the user, wherein the visible light frame being combined with its corresponding depth map frame provides color to a portion of the digital representation of the physical object; and
transmitting the digital representation of the physical object to an other device.
20. The computing system of claim 19 , wherein the other device is a second XR device worn by a second user and the digital representation of the physical object is displayed, via the second XR device, as a 3D model.