IP Library Granted Patent US 11,372,474
Granted Patent B2
US 11,372,474 · App. 16/919,983 · Granted Jun 28, 2022

Systems and methods for virtual artificial intelligence development and testing

Inventors: Richard Raymond Schweet (Kirkland, WA); Bendenetto Christopher Ruggiero (Cincinnati, OH); Kyle Robert Hartshorn (Cincinnati, OH); Gregory Ryan Sweeney (Cincinnati, OH); Kyle Dean Cypher (Houston, TX); Melissa Yenni Scharf (Cincinnati, OH); Emily Ann Meyer (Denver, CO); Alec Brenders Lisy (Cincinnati, OH); Jeremy David Jarrett (Cincinnati, OH); Matthew David Fye (Cincinnati, OH)
Assignee: SAEC/Kinetic Vision, Inc.
G06F3/011G06N5/04G06N20/00G06Q30/0201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,372,474
App. No.
16/919,983
Filed
Jul 2, 2020
Granted
Jun 28, 2022
Kind
B2
Art Unit
2626
USPC
345/156
Abstract

Systems and methods are provided to create training data, validate, deploy and test artificial intelligence (AI) systems in a virtual development environment, incorporating virtual spaces, objects, machinery, devices, subsystems, and actual human action and behavior.

Claims (51)

1. A computer implemented method, comprising:

provisioning, by a virtual environment computer system, a virtual environment to a first human actor through a first virtual reality interface of a first virtual reality device worn by the first human actor;

simultaneously provisioning, by the virtual environment computer system, the virtual environment to a second human actor through a second virtual reality interface of a second virtual reality device worn by the second human actor, wherein the first and second human actors are physically positioned in a studio, wherein the virtual environment is a digital twin virtual environment of a real-world physical environment, wherein the virtual environment comprises a virtual sensor modeled to generate a virtual output corresponding to an output of a real-world sensor;

tracking, by the virtual environment computer system, real-time physical motion of the first human actor within the studio;

tracking, by the virtual environment computer system, real-time physical motion of the second human actor within the studio;

presenting to the first human actor, at least a portion of a first humanoid avatar replicating the real-time physical motion of the first human actor within the digital twin virtual environment and a second humanoid avatar replicating the real-time physical motion of the second human actor within the digital twin virtual environment;

presenting to the second human actor, at least a portion of the second humanoid avatar replicating the real-time physical motion of the second human actor within the digital twin virtual environment and the first humanoid avatar replicating the real-time physical motion of the first human actor within the digital twin virtual environment;

collecting real-time information from the digital twin virtual environment with an artificial intelligence computing system, wherein the real-time information is based on the tracking of the real-time physical motion of the first and second human actors within the studio and the virtual output of the virtual sensor, wherein the artificial intelligence computing system is for use at the real-world physical environment that comprises the real-world sensor; and

prior to use of the artificial intelligence computing system at the real-world physical environment, testing the artificial intelligence computing system, wherein the testing is based on the tracking of the real-time physical motion of the first and second human actors within the studio.

2. The computer implemented method of claim 1 , wherein the tracking of the real-time physical motion of the first and second human actors comprises tracking the real-time physical motion of the first and second human actors using a motion capture system.

3. The computer implemented method of claim 2 , wherein the motion capture system comprises a first plurality of markers worn by the first human actor in the studio and a second plurality of markers worn by the second human actor in the studio.

4. The computer implemented method of claim 3 , wherein the first and second plurality of spatial markers are any of passive markers and active trackers.

5. The computer implemented method of claim 2 , wherein the motion capture system is a non-optical motion capture system.

6. The computer implemented method of claim 1 , wherein the tracking of the real-time physical motion of the first and second human actors comprises tracking real-time physical motion of appendages of the first human actor using a motion capture system and tracking real-time physical motion of appendages of the second human actor using a motion capture system.

7. The computer implemented method of claim 1 , wherein tracking of the real-time physical motion of the first and second human actors comprises tracking real-time physical motion of digits of the first human actor using a motion capture system and tracking real-time physical motion of digits of the second human actor using a motion capture system.

8. The computer implemented method of claim 1 , wherein a virtual object is presented to the virtual environments of the first and second human actors through the first and second virtual reality interfaces, respectively, wherein the virtual object is modeled from a real-world physical object associated with the real-world physical environment, wherein a physical training object is physically positioned in the studio with the human actor, wherein the physical training object is either a mock-up of the real-world physical object or the real-world physical object.

9. The computer implemented method of claim 8 , wherein the real-world physical environment is a retail environment, wherein the virtual object is modeled from a real-world retail product, and wherein the physical training object is a mock-up version of the real-world retail product.

10. The computer implemented method of claim 8 , wherein the real-world physical environment is a surgical environment, and wherein the virtual object is modeled from a real-world surgical instrument.

11. The computer implemented method of claim 10 , wherein the physical training object is an interactive mock-up of the surgical instrument, wherein the interactive mock-up of the surgical instrument comprises a physical actuator.

12. The computer implemented method of claim 11 , further comprising:

responsive to either of the first or second human actors actuating the physical actuator of the interactive mock-up, presenting to the first and second human actors an actuation of the virtual object in the digital twin virtual environment through the first and second virtual reality interfaces, respectively.

13. The computer implemented method of claim 1 , wherein the physical training object comprises a physical actuator moveable from a first position to a second position.

14. The computer implemented method of claim 13 , further comprising:

responsive to either of the first or second human actors moving the physical actuator of the physical training object from the first position to the second positon, presenting to the first and second human actors an actuation of the virtual object in the digital twin virtual environment through the first and second virtual reality interfaces, respectively.

15. The computer implemented method of claim 8 , wherein a plurality of virtual objects are presented in the digital twin virtual environment and a plurality of corresponding physical training objects are physically positioned within the studio.

16. The computer implemented method of claim 8 , wherein the virtual object is modeled from an object actual physical object that is in existence at the time of modeling.

17. The computer implemented method of claim 8 , wherein the virtual object is modeled from a proposed real-world physical object.

18. The computer implemented method of claim 1 , wherein testing the artificial intelligence computing system comprises:

varying a plurality of operational conditions of the digital twin virtual environment; and

identifying a performance metric based on the interaction of the first and second human actors.

19. The computer implemented method of claim 18 , subsequent to testing the artificial intelligence computing system, deploying the artificial intelligence computing system at the real-world physical environment for tracking of humans and human actions within the real-world physical environment.

20. The computer implemented method of claim 1 , wherein the real-world physical environment is an actual real-world physical environment that is in existence at the time of modeling.

21. The computer implemented method of claim 1 , wherein real-world physical environment is a proposed real-world physical environment.

22. A computer implemented method, comprising:

provisioning, by a virtual environment computer system, a virtual environment to a human actor through a virtual reality interface of a virtual reality device worn by the human actor, wherein the human actor is physically positioned in a studio, wherein the virtual environment is a digital twin virtual environment of a real-world physical environment, wherein the virtual environment comprises a virtual sensor modeled to generate a virtual output corresponding to an output of a real-world sensor, wherein a virtual object is presented to the human actor in the digital twin virtual environment through the virtual reality interface, wherein the virtual object is modeled from a real-world physical object associated with the real-world physical environment, wherein a physical training object is physically positioned in the studio with the human actor, and wherein the physical training object is either a mock-up of the real-world physical object or the real-world physical object;

tracking, by the virtual environment computer system, real-time physical motion of the human actor within the studio;

tracking, by the virtual environment computer system, an interaction of the human actor with the physical training object;

collecting real-time information from the digital twin virtual environment with an artificial intelligence computing system, wherein the real-time information is based on (i) the tracking of the real-time physical motion of the human actor within the studio, (ii) the virtual output of the virtual sensor, and (iii) interactions with the physical training object by the human actor, wherein the artificial intelligence computing system is for use at the real-world physical environment that comprises the real-world sensor; and

prior to use of the artificial intelligence computing system at the real-world physical environment, testing the artificial intelligence computing system, wherein the testing is based on the tracking of the real-time physical motion of the human actor within the studio and the tracking of the interaction of the human actor with the physical training object.

23. The computer implemented method of claim 22 , wherein the tracking of the real-time physical motion of the human actor comprises tracking the real-time physical motion of the human actor using a motion capture system.

24. The computer implemented method of claim 22 , wherein the real-world physical environment is any of a retail environment, a surgical environment, an industrial environment, a medical environment, a marine environment, a manufacturing environment, a military environments, and an outdoor environment.

25. The computer implemented method of claim 22 , wherein real-world physical environment is a surgical environment and the physical training object is an interactive mock-up of the surgical instrument, wherein the interactive mock-up of the surgical instrument comprises a physical actuator, and wherein the method further comprises:

responsive to the human actor actuating the physical actuator of the interactive mock-up, presenting to the human actor in the digital twin virtual environment through the virtual reality interface an actuation of the virtual object.

26. The computer implemented method of claim 22 , wherein the physical training object comprises a physical actuator moveable from a first position to a second position and wherein the method further comprises:

responsive to the human actor moving the physical actuator of the physical training object from the first position to the second positon, presenting to the human actor in the digital twin virtual environment through the virtual reality interface an actuation of the virtual object.

27. The computer implemented method of claim 22 , wherein testing an artificial intelligence computing system further comprises:

varying a plurality of operational conditions of the digital twin virtual environment; and

identifying a performance metric based on the interaction of the human actor with the virtual object.

28. The computer implemented method of claim 27 , subsequent to testing the artificial intelligence computing system, deploying the artificial intelligence computing system at the real-world physical environment for tracking of humans and human actions within the real-world physical environment.

29. The computer implemented method of claim 22 , wherein the digital twin virtual environment is a digital twin of a real-world physical environment that is in existence at the time of modeling or a digital twin of a proposed real-world physical environment.

30. The computer implemented method of claim 22 , wherein the virtual object is modeled from a real-world physical object that is in existence at the time of modeling or a proposed real-world physical object.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 16, 2020
From: FYE, MATTHEW DAVID
To: SAEC/KINETIC VISION, INC.
Reel/Frame 053228/0840 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 16, 2020
From: JARRETT, JEREMY DAVID
To: SAEC/KINETIC VISION, INC.
Reel/Frame 053228/0860 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 16, 2020
From: SCHWEET, RICHARD RAYMOND; RUGGIERO, BENDENETTO CHRISTOPHER; HARTSHORN, KYLE ROBERT; SWEENEY, GREGORY RYAN; CYPHER, KYLE DEAN; SCHARF, MELISSA YENNI; LISY, ALEC BRENDERS; MEYER, EMILY ANN
To: SAEC/KINETIC VISION, INC.
Reel/Frame 053228/0867 →
Continuity (2)
Provisional Application 62870326 · Jul 3, 2019
Related Publication 20210004076A1 · Jan 7, 2021
Cited By (8)
US 12,267,482 US 12,282,604 US 12,322,052 US 12,417,593 US 12,449,891 US 12,519,924 US 12,682,291 US 12,718,494