Granted Patent B2

US 12,361,657 · App. 17/946,562 · Granted Jul 15, 2025

Inventor: Donald Gerard Madden (Columbia, MD)

Assignee: ObjectVideo Labs, LLC

G06T19/20G06T19/006

Abstract

Claims (58)

1. A computer-implemented method comprising:

maintaining a database of types of extended reality objects and corresponding feature sets compatible for use with extended reality environments;

detecting one or more physical objects, for use with an extended reality environment;

determining, for each of the one or more physical objects and using the database of types of extended reality objects and corresponding feature sets compatible for use with extended reality environments, a corresponding feature set for the corresponding physical object, wherein at least one physical object has a different feature set than another physical object;

determining one or more candidate virtual objects for presentation in the extended reality environment;

determining, for each of the one or more candidate virtual objects, corresponding virtual feature sets;

determining whether a corresponding feature set for a physical object from the one or more physical objects satisfies a similarity criterion for a virtual feature set for a virtual object from the one or more candidate virtual objects;

determining whether to select the virtual object using a result of the determination whether the corresponding feature set for a physical object from the one or more physical objects satisfies the similarity criterion for a virtual feature set for the virtual object from the one or more candidate virtual objects;

selecting one or more virtual objects for presentation in the extended reality environment using the feature sets of the one or more physical objects;

generating, for each of the one or more physical objects and using the corresponding feature set and data for the one or more virtual objects, data for a representation of the physical object for use in the extended reality environment;

providing, to a user device, at least some of the data for the representation of the physical object to cause the user device to present at least a portion of the representation in the extended reality environment;

determining to change a presentation of the representation in the extended reality environment; and

in response to determining to change the presentation of the representation, controlling a physical position of the physical object using the change to the presentation of the representation in the extended reality environment.

2. The method of claim 1 , comprising generating the extended reality environment that includes the representation of the physical object.

3. The method of claim 1 , wherein:

detecting the one or more physical objects comprises determining whether a motorized object is available for representing objects in the extended reality environment; and

controlling the physical position of the physical object comprises sending, to the motorized object, instructions to cause the motorized object to move to a physical location that corresponds to a virtual location of the representation in the extended reality environment in response to determining that the motorized object is available for representing objects in the extended reality environment.

4. The method of claim 1 , wherein generating the data for the representation comprises generating a first representation i) of a first physical object from the one or more physical objects ii) that has the same type as the first physical object.

5. The method of claim 1 , wherein generating the data for the representation comprises generating a first representation i) of a first physical object from the one or more physical objects ii) that has a different type than the first physical object.

6. The method of claim 1 , wherein determining to change the presentation of the representation comprises receiving, through the user device, data indicating a user-input adjustment to the representation in the extended reality environment.

7. The method of claim 1 , comprising changing the presentation of the representation in the extended reality environment substantially concurrently with controlling the physical position of the physical object that corresponds to the representation.

8. A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

detecting one or more physical objects for use with an extended reality environment;

determining, for each of the one or more physical objects, a corresponding feature set;

selecting one or more virtual objects for presentation in the extended reality environment using the feature sets of the one or more physical objects, the selecting comprising:

determining one or more candidate virtual objects for presentation in the extended reality environment;

determining, for each of the one or more candidate virtual objects, corresponding virtual feature sets;

determining whether a corresponding feature set for a physical object from the one or more physical objects satisfies a similarity criterion for a virtual feature set for a virtual object from the one or more candidate virtual objects; and

determining whether to select the virtual object using a result of the determination whether the corresponding feature set for a physical object from the one or more physical objects satisfies the similarity criterion for a virtual feature set for the virtual object from the one or more candidate virtual objects;

generating, for each of the one or more physical objects and using the corresponding feature set and data for the one or more virtual objects, data for a representation of the physical object for use in the extended reality environment;

providing, to a user device, at least some of the data for the representation of the physical object to cause the user device to present at least a portion of the representation in the extended reality environment;

determining to change a presentation of the representation in the extended reality environment; and

in response to determining to change the presentation of the representation, controlling a physical position of the physical object using the change to the presentation of the representation in the extended reality environment.

9. The system of claim 8 , the operations comprising generating the extended reality environment that includes the representation of the physical object.

10. The system of claim 8 , wherein:

detecting the one or more physical objects comprises determining whether a motorized object is available for representing objects in the extended reality environment; and

controlling the physical position of the physical object comprises, sending, to the motorized object, instructions to cause the motorized object to move to a physical location that corresponds to a virtual location of the representation in the extended reality environment in response to determining that the motorized object is available for representing objects in the extended reality environment.

11. The system of claim 8 , wherein generating the data for the representation comprises generating a first representation i) of a first physical object from the one or more physical objects ii) that has the same type as the first physical object.

12. The system of claim 8 , wherein generating the data for the representation comprises generating a first representation i) of a first physical object from the one or more physical objects ii) that has a different type than the first physical object.

13. The system of claim 8 , wherein determining to change the presentation of the representation comprises receiving, through the user device, data indicating a user-input adjustment to the representation in the extended reality environment.

14. The system of claim 8 , the operations comprising changing the presentation of the representation in the extended reality environment substantially concurrently with controlling the physical position of the physical object that corresponds to the representation.

15. A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

detecting one or more physical objects for use with an extended reality environment having a first extended reality scenario;

determining, for each of the one or more physical objects, a corresponding feature set;

determining, using the feature sets of the one or more physical objects, that none of the feature sets include a feature required for presentation of the first extended reality scenario;

altering the first extended reality scenario to a second extended reality scenario using the feature sets for the determined one or more physical objects and in response to determining that none of the feature sets include a feature required for presentation of the first extended reality scenario;

selecting one or more virtual objects for presentation in the second extended reality scenario for the extended reality environment using the feature sets of the one or more physical objects;

generating, for each of the one or more physical objects and using the corresponding feature set and data for the one or more virtual objects, data for a representation of the physical object for use in the extended reality environment;

providing, to a user device, at least some of the data for the representation of the physical object to cause the user device to present at least a portion of the representation in the extended reality environment;

determining to change a presentation of the representation in the extended reality environment; and

in response to determining to change the presentation of the representation, controlling a physical position of the physical object using the change to the presentation of the representation in the extended reality environment.

16. The method of claim 1 , wherein the corresponding features set for the one or more physical objects includes one or more a tilt ability, a movement ability, a maximum speed, or an acceleration rate.

17. The system of claim 8 , wherein the corresponding features set for the one or more physical objects includes one or more a tilt ability, a movement ability, a maximum speed, or an acceleration rate.

18. The computer storage medium of claim 15 , wherein the corresponding features set for the one or more physical objects includes one or more a tilt ability, a movement ability, a maximum speed, or an acceleration rate.

19. The method of claim 1 , wherein:

detecting the one or more physical objects comprises determining whether a motorized object is available for representing objects in the extended reality environment and to skip changing the presentation of the representation of the extended reality environment; and

controlling the physical position of the physical object comprises sending, to the motorized object, instructions to cause the motorized object to move to a physical location that corresponds to a virtual location of the representation in the extended reality environment in response to determining that the motorized object is available for representing objects in the extended reality environment and to skip changing the presentation of the representation of the extended reality environment.

20. The computer storage medium of claim 15 , wherein generating the data for the representation comprises generating a first representation i) of a first physical object from the one or more physical objects ii) that has a different type than the first physical object.

Assignments (1)

Continuity (2)

References Cited (13)