Granted Patent B2

US 12,374,015 · App. 17/711,893 · Granted Jul 29, 2025

Inventor: Geoff Wedig (San Mateo, CA)

Assignee: Sony Interactive Entertainment LLC

G06T13/40A63F13/57G06T17/20G06V10/774G06V40/168G06V40/172G06V40/174

Abstract

Claims (32)

1. A method comprising:

instructing multiple simulated characters to generate multiple facial expressions using input label value files (iLVFs), the multiple simulated characters each having different facial features or physical attributes;

for each of the multiple facial expressions of the multiple simulated characters, capturing mesh data of the simulated character using a virtual camera to generate three dimensional (3D) depth data of a face of the simulated character, the 3D depth data being output as mesh files corresponding to frames captured by the virtual camera;

processing the iLVFs and the mesh data to train a model regarding correspondences between the iLVFs and the mesh data for multiple simulated characters; and

generating, by the model, output label file values (oLVFs) for animating a particular facial expression of a game character based on receiving input mesh files for a non-simulated, human actor performing the particular facial expression.

2. The method of claim 1 , wherein the mesh data is processed in time coordination with the iLVFs.

3. The method of claim 1 , comprising transmitting wherein the OLVFs to a game engine for animating the particular facial expression of the game character.

4. The method of claim 3 , wherein the oLVFs are used by the game engine to activate muscles on a face of the game character to generate the facial expressions.

5. The method of claim 1 , wherein the iLVFs are used to instruct the simulated character to activate muscles on a face of the simulated character to generate the facial expressions of the simulated characters.

6. The method of claim 1 , wherein the mesh data captured by the virtual camera corresponds to the facial expressions generated by the simulated characters.

7. The method of claim 1 , wherein the oLVFs includes a plurality of values that correspond to features on a face of the human actor, said plurality of values being used for animating the facial expression of the game character.

8. The method of claim 7 , wherein the plurality of values are configured to cause muscle activation in respective areas on a face of the game character.

9. The method of claim 1 , wherein key frames are identified from the mesh data, said key frames are processed to train the model.

10. The method of claim 1 , wherein the oLVFs include an emotion type, said emotion type is produced by the human actor to generate the input mesh files.

11. The method of claim 1 , wherein the model is configured to identify features from the mesh data and the iLVFs to classify attributes of the mesh data and the iLVFs, the attributes being used for generating the oLVFs corresponding to the input mesh files.

12. The method of claim 1 , wherein the human actor and the game character shares physical attributes.

13. The method of claim 1 , wherein the game character is an avatar representing the human actor.

14. A method for generating label values for facial expressions of a game character using three-dimensional (3D) image capture, comprising:

accessing a model that is trained using inputs captured of multiple simulated characters using a virtual camera, the multiple simulated characters having different facial features or physical attributes;

the inputs captured additionally include input label value files (iLVFs) that are used to generate facial expressions of the multiple simulated characters;

the inputs further include mesh data of a face of the multiple simulated characters, the mesh data representing three-dimensional (3D) depth data of the face;

the model being trained by processing the iLVFs and the mesh data for the multiple simulated characters, the training of the model is configured to learn correspondences between the iLVFs and the mesh data;

capturing mesh files that include mesh data of a face of a human actor, the mesh files being provided as input queries to the model to generate one or more output label value files (oLVFs); and

animating the facial expressions of the game character presented in a game processed by a game engine based at least on the one or more oLVFs.

15. The method of claim 14 , wherein the iLVFs and the mesh data is processed in time coordination such that correspondences between the iLVFs and the mesh data are learned by the model.

16. The method of claim 14 , wherein the LVFs are used by the game engine to activate muscles on a face of the game character to generate the facial expressions of the game character.

17. The method of claim 14 , wherein the iLVFs are used to instruct the simulated character to activate muscles on the face of the simulated characters to generate the facial expressions of the simulated characters.

18. The method of claim 14 , wherein the oLVFs includes a plurality of values that correspond to features on the face of the human actor, said plurality of values being used for animating the facial expression of the game character.

19. The method of claim 18 , wherein the plurality of values is configured to cause muscle activation in respective areas on the face of the game character.

20. The method of claim 14 , wherein the model is configured to identify features from the mesh data and the iLVFs to classify attributes of the mesh data and the iLVFs, the attributes being used for generating the oLVFs corresponding to the captured mesh files.

21. The method of claim 14 , wherein the human actor and the game character do not resemble one another.

22. The method of claim 14 , wherein the human actor and the multiple simulated characters do not resemble one another.

Assignments (1)

Continuity (2)

References Cited (37)