Granted Patent B2

US 11,494,995 · App. 17/083,808 · Granted Nov 8, 2022

Inventor: Robert David Berkebile (Phoenix, MD)

Assignee: Magic Leap, Inc.

G06T19/006G06T17/20

Abstract

Claims (61)

1. A computer implemented method, comprising: determining a three-dimensional model for at least a portion of a physical environment in which a user is located;

presenting, by a spatial computing system, a mixed-reality representation to the user;

determining a first object model for a first object in the mixed-reality representation;

updating, by the spatial computing system, the mixed-reality representation into an updated mixed-reality representation that reflects an interaction pertaining to the first object; and

recognizing one or more second objects in the at least the portion of the physical environment or in the mixed-reality representation, wherein the recognizing the one or more second objects comprises:

identifying one or more first object recognizers and one or more second object recognizers, wherein an object recognizer of the one or more first object recognizers and one or more second object recognizers is pre-configured to recognize a type of object or a specific object;

performing a multi-level recognition process to the one or more second objects, wherein the multi-level recognition process comprises a first level of applying the one or more first object recognizers and a second level of applying the one or more second object recognizers, and the first level is performed before the second level is performed;

pruning at least one second object from the one or more second objects at least by applying the one or more first object recognizers to the one or more second objects; and

recognize a second object in the one or more second objects at least by applying the one or more second object recognizers to one or more remaining second objects, wherein the one or more remaining second objects do not include the at least one second object.

2. The computer implemented method of claim 1 , determining the three-dimensional model comprising:

recognizing one or more physical objects in the at least the portion of the physical environment; and

decomposing the at least the portion of the physical environment into one or more sets of voxels.

3. The computer implemented method of claim 2 , determining the three-dimensional model further comprising:

meshing at least one voxel of the one or more sets of voxels with two-dimensional and/or three-dimensional mesh; and

representing the at least the portion with the two-dimensional and/or three-dimensional mesh.

4. The computer implemented method of claim 3 , determining the three-dimensional model further comprising:

replacing a physical object of the one or more physical objects with a physical object model.

5. The computer implemented method of claim 1 , further comprising:

identifying one or more properties for the second object based at least in part upon a type of the second object; and

associating the one or more properties for the second object, wherein the one or more properties comprise one or more moduli, a density, a mass, a weight, a friction characteristic, or an electrical characteristic of the second object.

6. The computer implemented method of claim 1 , wherein the interaction comprises a user-to-object interaction between the user and the first object or an object-to-object interaction between the first object and a different object, and the different object comprises a virtual object or a physical object.

7. The computer implemented method of claim 1 , wherein updating the mixed-reality representation into the updated mixed-reality representation comprises at least one of:

performing a physics-based behavior modeling for at least the first object model;

performing object interaction modeling for the interaction and at least the first object model; or

performing object soft locking for the first object or a separate object in the mixed-reality representation or the updated mixed-reality representation.

8. A system, comprising:

a processor;

a headset comprising at least one projector and configured to project virtual content of a virtual object to a user; and

a non-transitory computer accessible storage medium storing thereupon a sequence of instructions which, when executed by the at least one processor, causes the processor to perform a set of acts, the set of acts comprising:

determining a three-dimensional model for at least a portion of a physical environment in which a user is located;

presenting, by a spatial computing system, a mixed-reality representation to the user;

determining a first object model for a first object in the mixed-reality representation;

updating, by the spatial computing system, the mixed-reality representation into an updated mixed-reality representation that reflects an interaction pertaining to the first object; and

the sequence of instructions further comprising instructions which, when executed by the processor, cause the processor to perform the set of acts, the set of acts further comprising:

identifying one or more first object recognizers and one or more second object recognizers, wherein an object recognizer of the one or more first object recognizers and one or more second object recognizers is pre-configured to recognize a type of object or a specific object;

recognizing one or more second objects in the at least the portion of the physical environment or in the mixed-reality representation by performing a multi-level recognition process to the one or more second objects, wherein the multi-level recognition process comprises a first level of applying the one or more first object recognizers and a second level of applying the one or more second object recognizers, and the first level is performed before the second level is performed;

pruning at least one second object from the one or more second objects at least by applying the one or more first object recognizers to the one or more second objects;

recognizing a second object in the one or more second objects at least by applying the one or more second object recognizers to one or more remaining second objects, wherein the one or more remaining second objects do not include the at least one second object;

identifying one or more properties for the second object based at least in part upon a type of the second object; and

associating the one or more properties for the second object, wherein the one or more properties comprise one or more moduli, a density, a mass, a weight, a friction characteristic, or an electrical characteristic of the second object.

9. The system for claim 8 , the set of acts further comprising:

recognizing one or more physical objects in the at least the portion of the physical environment;

decomposing the at least the portion of the physical environment into one or more sets of voxels;

meshing at least one voxel of the one or more sets of voxels with two-dimensional and/or three-dimensional mesh; and

representing the at least the portion with the two-dimensional and/or three-dimensional mesh; and

replacing a physical object of the one or more physical objects with a physical object model.

10. A computer program product comprising a non-transitory computer readable storage medium having stored thereupon a sequence of instructions which, when executed by a processor, causes the processor to perform a set of acts, the set of acts comprising:

determining a three-dimensional model for at least a portion of a physical environment in which a user is located;

presenting, by a spatial computing system, a mixed-reality representation to the user;

determining a first object model for a first object in the mixed-reality representation;

updating, by the spatial computing system, the mixed-reality representation into an updated mixed-reality representation that reflects an interaction pertaining to the first object;

identifying one or more first object recognizers and one or more second object recognizers, wherein an object recognizer of the one or more first object recognizers and one or more second object recognizers is pre-configured to recognize a type of object or a specific object;

recognizing one or more second objects in the at least the portion of the physical environment or in the mixed-reality representation at least by performing a multi-level recognition process to the one or more second objects, wherein the multi-level recognition process comprises a first level of applying the one or more first object recognizers and a second level of applying the one or more second object recognizers, and the first level is performed before the second level is performed;

pruning at least one second object from the one or more second objects at least by applying the one or more first object recognizers to the one or more second objects;

recognizing a second object in the one or more second objects at least by applying the one or more second object recognizers to one or more remaining second objects, wherein the one or more remaining second objects do not include the at least one second object, identifying one or more properties for the second object based at least in part upon a type of the second object; and associating the one or more properties for the second object, wherein the one or more properties comprise one or more moduli, a density, a mass, a weight, a friction characteristic, or an electrical characteristic of the second object.

11. The computer program product of claim 10 , the set of acts further comprising:

recognizing one or more physical objects in the at least the portion of the physical environment;

decomposing the at least the portion of the physical environment into one or more sets of voxels;

meshing at least one voxel of the one or more sets of voxels with two-dimensional and/or three-dimensional mesh;

representing the at least the portion with the two-dimensional and/or three-dimensional mesh; and

replacing a physical object of the one or more physical objects with a physical object model.

Assignments (3)

Continuity (3)

Cited By (11)

US 12,277,652 US 12,307,600 US 12,347,124 US 12,355,712 US 12,367,626 US 12,394,166 US 12,423,855 US 12,469,194 US 12,482,172 US 12,482,208 US 12,499,574