IP Library › Granted Patent US 12,296,265
Granted Patent B1
US 12,296,265 · App. 18/407,686 · Granted May 13, 2025

Speech prosody prediction in video games

Inventors: Kilol Gupta (Redwood City, CA); Zahra Shakeri (Newark, CA); Gordon Durity (Surrey, CA); Mohsen Sardari (Burlingame, CA); Harold Chaput (Castro Valley, CA); Navid Aghdaie (San Jose, CA)
Assignee: ELECTRONIC ARTS INC.
A63F13/54G06F3/16G06N3/08G10L13/02G10L19/04G10L25/30G10L25/51A63F2300/6081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,296,265
App. No.
18/407,686
Granted
May 13, 2025
Kind
B1
Abstract

This specification describes a computer-implemented method of generating context-dependent speech audio in a video game. The method comprises obtaining contextual information relating to a state of the video game. The contextual information is inputted into a prosody prediction module. The prosody prediction module comprises a trained machine learning model which is configured to generate predicted prosodic features based on the contextual information. Input data comprising the predicted prosodic features and speech content data associated with the state of the video game is inputted into a speech audio generation module. An encoded representation of the speech content data dependent on the predicted prosodic features is generated using one or more encoders of the speech audio generation module. Context-dependent speech audio is generated, based on the encoded representation, using a decoder of the speech audio generation module.

Claims (69)

1. A computer-implemented method of generating context-dependent speech audio in a video game, the method comprising:

enabling, by at least one processor of a computing device, gameplay of the video game;

determining, by a video game engine of the video game on the at least one processor, an in-game event for which context-dependent speech audio is to be generated during the gameplay of the video game, wherein the in-game event includes an action performed by a character of the video game;

obtaining, by the video game engine of the video game, contextual information and speech content data relating to a current state of the gameplay;

requesting, by the video game engine of the video game, the context-dependent speech audio from a speech audio generator of the video game;

generating, by the speech audio generator responsive to the request, the context-dependent speech audio by:

inputting the contextual information relating to the current state of the gameplay into a prosody prediction model, wherein the prosody prediction model comprises a trained machine learning model which is configured to generate predicted prosodic features based on the contextual information;

generating, by the prosody prediction model, predicted prosodic features from the input contextual information;

inputting, into a speech audio generation model, input data comprising:

at least the predicted prosodic features; and

the speech content data relating to the current state of the gameplay;

generating, using one or more encoders of the speech audio generation model, an encoded representation of the speech content data dependent on the predicted prosodic features;

decoding, using a decoder of the speech audio generation model, the encoded representation to generate the context-dependent speech audio; and

causing, by the video game engine of the video game, the context-dependent speech audio that matches the current state of the video game to be played among the gameplay of the in-game event.

2. The computer-implemented method of claim 1 , wherein the one or more encoders comprise a prosody encoder configured to generate an encoded representation of the predicted prosodic features, and a speech content encoder configured to generate the encoded representation of the speech content data based on the encoded representation of the predicted prosodic features.

3. The computer-implemented method of claim 1 , wherein the video game is a sports video game, wherein obtaining the contextual information relating to the current state of the video game comprises determining contextual information relating to an in-progress match of the sports video game.

4. The computer-implemented method of claim 3 , wherein the contextual information relating to the in-progress match of the sports video game comprises determining one or more of:

statistics relating to one or more teams playing in the match;

statistics relating to one or more players playing in the match;

statistics relating to a current status of the match; and

the type of sport being played in the match.

5. The computer-implemented method of claim 1 , wherein the contextual information includes the speech content data associated with the current state of the video game.

6. The computer-implemented method of claim 1 , wherein the input data further comprises speaker identifier data for a speaker of the generated speech audio.

7. A non-transitory computer-readable medium containing instructions, which when executed by one or more processors, causes the one or more processors to perform a method comprising:

enabling, by at least one processor of a computing device, gameplay of a video game;

determining, by a video game engine of the video game on the at least one processor, an in-game event for which context-dependent speech audio is to be generated during the gameplay of the video game, wherein the in-game event includes an action performed by a character of the video game;

obtaining, by the video game engine of the video game, contextual information and speech content data relating to a current state of the gameplay;

requesting, by the video game engine of the video game, the context-dependent speech audio from a speech audio generator of the video game;

generating, by the speech audio generator responsive to the request, the context-dependent speech audio by:

inputting the contextual information relating to the current state of the gameplay into a prosody prediction model, wherein the prosody prediction model comprises a trained machine learning model which is configured to generate predicted prosodic features based on the contextual information;

generating, by the prosody prediction model, predicted prosodic features from the input contextual information;

inputting, into a speech audio generation model, input data comprising:

at least the predicted prosodic features; and

the speech content data relating to the current state of the gameplay;

generating, using one or more encoders of the speech audio generation model, an encoded representation of the speech content data dependent on the predicted prosodic features;

decoding, using a decoder of the speech audio generation model, the encoded representation to generate the context-dependent speech audio; and

causing, by the video game engine of the video game, the context-dependent speech audio that matches the current state of the video game to be played among the gameplay of the in-game event.

8. The non-transitory computer-readable medium of claim 7 , wherein the speech audio generation model includes a synthesizer.

9. The non-transitory computer-readable medium of claim 8 , wherein the speech content data comprises a plurality of speech content segments at a plurality of respective time steps and wherein inputting, into the speech audio generation model, the input data comprising the predicted prosodic features and the speech content data comprises generating, as output of a speech content encoder of the synthesizer, a speech content encoding for each time step of one or more time steps of the speech content data.

10. The non-transitory computer-readable medium of claim 9 , wherein generating predicted prosodic features comprises generating predicted prosodic features for each time step of the one or more time steps of the speech content data.

11. The non-transitory computer-readable medium of claim 10 , wherein inputting, into the speech audio generator, the input data comprising the predicted prosodic features and the speech content data comprises combining, for each time step of the one or more time steps, the speech content encoding and the predicted prosodic features of the time step.

12. A computer-implemented method of generating context-dependent speech audio in a video game, the method comprising:

enabling, by at least one processor of a computing device, gameplay of the video game comprising requesting, by the at least one processor of the computing device, video game content from a video game server while a user is playing the video game;

determining, by a video game engine of the video game on the at least one processor, an in-game event for which context-dependent speech audio is to be generated during the gameplay of the video game;

obtaining, by the video game engine of the video game, contextual information and speech content data relating to a current state of the gameplay;

requesting, by the video game engine of the video game, the context-dependent speech audio from a speech audio generator of the video game;

generating, by the speech audio generator responsive to the request, the context-dependent speech audio based upon processing the contextual information and speech content data relating to the current state of the gameplay by one or more machine learning models; and

causing, by the video game engine of the video game, the context-dependent speech audio that matches the current state of the video game to be played among the gameplay of the in-game event.

13. The computer-implemented method of claim 12 , wherein generating, by the speech audio generator responsive to the request, the context-dependent speech audio based upon processing the contextual information and speech content data relating to the current state of the gameplay by the one or more machine learning models comprises:

generating, using the one or more machine learning models, predicted prosodic features based upon the contextual information.

14. The computer-implemented method of claim 13 , wherein generating, by the speech audio generator responsive to the request, the context-dependent speech audio based upon processing the contextual information and speech content data relating to the current state of the gameplay by the one or more machine learning models comprises:

generating, using the one or more machine learning models, the speech content data based upon the predicted prosodic features.

15. The computer-implemented method of claim 12 , wherein the video game is a sports video game, wherein obtaining the contextual information relating to the current state of the video game comprises determining contextual information relating to an in-progress match of the sports video game.

16. The computer-implemented method of claim 15 , wherein the contextual information relating to the in-progress match of the sports video game comprises determining one or more of:

statistics relating to one or more teams playing in the match;

statistics relating to one or more players playing in the match;

statistics relating to a current status of the match; and

the type of sport being played in the match.

17. The computer-implemented method of claim 12 , wherein the contextual information includes the speech content data associated with the current state of the video game.

18. The computer-implemented method of claim 12 , wherein an input data further comprises speaker identifier data for a speaker of the generated speech audio.

19. A non-transitory computer-readable medium containing instructions, which when executed by one or more processors, causes the one or more processors to perform a method comprising:

enabling, by at least one processor of a computing device, gameplay of a video game comprising requesting, by the at least one processor of the computing device, video game content from a video game server while a user is playing the video game;

determining, by a video game engine of the video game on the at least one processor, an in-game event for which context-dependent speech audio is to be generated during the gameplay of the video game;

obtaining, by the video game engine of the video game, contextual information and speech content data relating to a current state of the gameplay;

requesting, by the video game engine of the video game, the context-dependent speech audio from a speech audio generator of the video game;

generating, by the speech audio generator responsive to the request, the context-dependent speech audio based upon processing the contextual information and speech content data relating to the current state of the gameplay by one or more machine learning models; and

causing, by the video game engine of the video game, the context-dependent speech audio that matches the current state of the video game to be played among the gameplay of the in-game event.

20. The non-transitory computer-readable medium of claim 19 , wherein generating, by the speech audio generator responsive to the request, the context-dependent speech audio based upon processing the contextual information and speech content data relating to the current state of the gameplay by the one or more machine learning models comprises:

generating, using the one or more machine learning models, predicted prosodic features based upon the contextual information.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 11, 2024
From: GUPTA, KILOL; SHAKERI, ZAHRA; DURITY, GORDON; SARDARI, MOHSEN; CHAPUT, HAROLD; AGHDAIE, NAVID
To: ELECTRONIC ARTS INC.
Reel/Frame 066092/0231 →
Continuity (1)
Continuation 16953801 · Nov 20, 2020
References Cited (44)
US 10249289B2 · Chun · 2019 [cited by examiner]
US 10600404B2 · Garcia · 2020 [cited by examiner]
US 11605388B1 · Gupta · 2023 [cited by examiner]
US 20150012277A1 · Stephens, Jr. · 2015 [cited by examiner]
US 20160189704A1 · Suzuki · 2016 [cited by examiner]
US 20180221769A1 · Kosai · 2018 [cited by examiner]
US 20190043472A1 · Garcia · 2019 [cited by examiner]
US 20190043474A1 · Kingsbury · 2019 [cited by examiner]
US 20200394998A1 · Kim · 2020 [cited by examiner]
US 20200410241A1 · Sundareson · 2020 [cited by examiner]
US 20210097976A1 · Chicote · 2021 [cited by examiner]
US 20210233534A1 · Nadig · 2021 [cited by examiner]
US 20220051654A1 · Finkelstein · 2022 [cited by examiner]
US 20230075891A1 · Zheng · 2023 [cited by examiner]
Arik, Sercan O., et al. “Neural Voice Cloning with a Few Samples.” arXiv preprint arXiv:1802.06006 (2018), Retrieved from https://arxiv.org/abs/1802.06006v3. [cited by applicant]
Jia, Ye, et al. “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis.” arXiv preprint arXiv:1806.04558 (2018), Retrieved from https://arxiv.org/abs/1806.04558. [cited by applicant]
Polyak, Adam, et al. “TTS skins: Speaker conversion via ASR.” arXiv preprint arXiv:1904.08983 (2019), Retrieved from https://arxiv.org/abs/1904.08983. [cited by applicant]
Sun, Lifa, et al. “Voice conversion using deep bidirectional long short-term memory based recurrent neural networks.” 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015. ,… [cited by applicant]
Kameoka, Hirokazu, et al. “StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks.” arXiv preprint arXiv:1806.02169 (2018), Retrieved from https://arxiv.org/abs/1806.02169. [cited by applicant]
Liu, Li-Juan, et al. “WaveNet Vocoder with Limited Training Data for Voice Conversion.” Interspeech. 2018, Retrieved from https://www.isca-speech.org/archive/Interspeech_2018/pdfs/1190.pdf. [cited by applicant]
Narayanan, Praveen, et al. “Hierarchical sequence to sequence voice conversion with limited data.” arXiv preprint arXiv:1907.07769 (2019), Retrieved from https://arxiv.org/abs/1907.07769. [cited by applicant]
Zhang, Mingyang, et al. “Joint training framework for text-to-speech and voice conversion using multi-source tacotron and wavenet.” arXiv preprint arXiv:1903.12389 (2019), Retrieved from https://arxiv.org/abs/1903.12389. [cited by applicant]
Huang, Wen-Chin, et al. “Voice transformer network: Sequence-to-sequence voice conversion using transformer with text-to-speech pretraining.” arXiv preprint arXiv:1912.06813 (2019), Retrieved from https://arxiv.org/abs/… [cited by applicant]
Luong, Hieu-Thi, and Junichi Yamagishi. “Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech.” arXiv preprint arXiv:1909.06532 (2019), Retrieved from https://arxiv.org/abs/1909.06532. [cited by applicant]
Kim, Tae-Ho, et al. “Emotional Voice Conversion using multitask learning with Text-to-speech.” arXiv preprint arXiv:1911.06149 (2019), Retrieved from: https://arxiv.org/abs/1911.06149. [cited by applicant]
Ren, Yi, et al. “FastSpeech 2: Fast and High-Quality End-to-End Text-to-Speech.” arXiv preprint arXiv:2006.04558 (2020), Retrieved from: https://arxiv.org/abs/2006.04558. [cited by applicant]
Skerry-Ryan, R. J., et al. “Towards end-to-end prosody transfer for expressive speech synthesis with tacotron.” arXiv preprint arXiv:1803.09047 (2018), Retrieved from: https://arxiv.org/abs/1803.09047. [cited by applicant]
Theune, Mariët, et al. “Generating expressive speech for storytelling applications.” IEEE Transactions on Audio, Speech, and Language Processing 14.4 (2006): 1137-1144., Retrieved from: https://ieeexplore.IEEE.org/abstr… [cited by applicant]
Gibiansky, Andrew, et al. “Deep voice 2: Multi-speaker neural text-to-speech.” Advances in neural information processing systems. 2017, Retrieved from: http://papers.nips.cc/paper/6889-deep-voice-2-multi-speaker-neural-… [cited by applicant]
Shen, Jonathan, et al. “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.” 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, Retrieved from: … [cited by applicant]
Wu, Xixin, et al. “Rapid Style Adaptation Using Residual Error Embedding for Expressive Speech Synthesis.” Interspeech. 2018, Retrieved from: http://www1.se.cuhk.edu.hk/˜hccl/publications/pub/2018_201809_INTERSPEECH_Xix… [cited by applicant]
Zen, Heiga, et al. “The HMM-based speech synthesis system (HTS) version 2.0.” SSW. 2007, Retrieved from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.69.5302&rep=rep1&type=pdf. [cited by applicant]
Kalchbrenner, Nal, et al. “Efficient Neural Audio Synthesis.” International Conference on Machine Learning. 2018, Retrieved from: http://proceedings.mlr.press/v80/kalchbrenner18a.html. [cited by applicant]
Eyben, Florian, et al. “Unsupervised clustering of emotion and voice styles for expressive TTS.” 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2012, Retrieved from: https:… [cited by applicant]
Wang, Yuxuan, et al. “Tacotron: Towards end-to-end speech synthesis.” arXiv preprint arXiv:1703.10135 (2017), Retrieved from: https://arxiv.org/abs/1703.10135. [cited by applicant]
Wang, Yuxuan, et al. “Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis.” ICML. 2018, Retrieved from: https://openreview.net/forum?id=Hy4s90-OZr. [cited by applicant]
Akuzawa, Kei, Yusuke Iwasawa, and Yutaka Matsuo. “Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder.” Proc. Interspeech 2018 (2018): 3067-3071, Retrieved from: https://www.isca-speech.org… [cited by applicant]
Taigman, Yaniv, et al. “VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop.” International Conference on Learning Representations. 2018, Retrieved from: https://openreview.net/forum?id=SkFAWax0-&noteld=SkFAW… [cited by applicant]
Hsu, Wei-Ning, et al. “Hierarchical Generative Modeling for Controllable Speech Synthesis.” International Conference on Learning Representations. 2018, Retrieved from: https://openreview.net/forum?id=rygkk305YQ. [cited by applicant]
Kenter, Tom, et al. “CHiVE: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network.” International Conference on Machine Learning. 2019, Retrieved from: htt… [cited by applicant]
Lee, Younggun, and Taesu Kim. “Robust and fine-grained prosody control of end-to-end speech synthesis.” ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, Ret… [cited by applicant]
Klimkov, Viacheslav, et al. “Fine-Grained Robust Prosody Transfer for Single-Speaker Neural Text-To-Speech.” Proc. Interspeech 2019 (2019): 4440-4444, Retrieved from: https://www.isca-speech.org/archive/Interspeech_2019… [cited by applicant]
Daniel, Povey, et al. “The Kaldi speech recognition toolkit.” IEEE 2011 workshop on automatic speech recognition and understanding. No. EPFL-CONF-192584. 2011, Retrieved from: https://www.fit.vut.cz/research/product/304… [cited by applicant]
Ghahremani, Pegah, et al. “A pitch extraction algorithm tuned for automatic speech recognition.” 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2014, Retrieved from: https:… [cited by applicant]
Cited By (1)
US 12,605,109