IP Library Granted Patent US 11,501,764
Granted Patent B2
US 11,501,764 · App. 16/408,887 · Granted Nov 15, 2022

Apparatus for media entity pronunciation using deep learning

Inventor: Daniel Bromand (Stockholm, SE)
Assignee: Spotify AB
G10L15/187G06N20/00G10L13/00G10L15/063G10L15/16G10L15/22
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,501,764
App. No.
16/408,887
Granted
Nov 15, 2022
Kind
B2
Abstract

Methods, systems, and related products for voice-enabled computer systems are described. A machine-learning model is trained to produce pronunciation output based on text input. The trained machine-learning model is used to produce pronunciation data for text input even where the text input includes numbers, punctuation, emoji, or other non-letter characters. The machine-learning model is further trained based on real-world data from users to improve pronunciation output.

Claims (48)

1. A computer-implemented method comprising:

receiving text data including at least one character that is a non-letter character having a standard pronunciation;

providing the text data as input into a trained machine-learning model;

receiving as output from the trained machine-learning model, output data indicative of a pronunciation of the text data, the pronunciation of the non-letter character being different from the standard pronunciation;

wherein the trained machine learning model is trained using an utterance from a user, wherein the training includes:

processing the utterance with a speech-to-text engine to identify within the utterance at least a first and a second portion;

conducting a search to find a match to the first portion in a metadata database, the match having associated data, wherein the metadata database includes metadata which describes media content, and the metadata database includes at least the following categories of metadata: song titles, artist names, and albums, wherein the match and the associated data are each associated with different categories of the metadata in the metadata database;

analyzing the associated data to identify an associated portion in the associated data that has a similarity to the second portion, wherein the associated portion includes a non-letter character; and

training the machine-learning model with the associated portion as training input data and the second portion as training output data.

2. The method of claim 1 , wherein training data includes text-based words and associated correct pronunciations.

3. The method of claim 2 , wherein at least one of the text-based words includes a character that is a non-letter character.

4. The method of claim 2 , wherein training the machine-learning model includes training the machine-learning model to produce pronunciation output based on text input.

5. The method of claim 1 , wherein the similarity between the second and associated portions is determined to pass a predetermined threshold.

6. The method of claim 5 , wherein the similarity is determined based on a number of characters that are the same between the second and associated portions passing the predetermined threshold.

7. The method of claim 6 , further comprising:

identifying a media content item associated with the match; and

initiating playback of the media content item.

8. The method of claim 1 , wherein training data includes the training output data based on a standardized phonetic representation of a spoken language.

9. The method of claim 8 , wherein the standardized phonetic representation is an International Phonetic Alphabet (IPA), a Speech Assessment Methods Phonetic Alphabet (SAMPA), an Extended SAMPA (X-SAMPA), or a Speech Synthesis Markup Language (SSML).

10. The method of claim 8 , wherein the training output data is formatted as a vector representation.

11. The method of claim 1 , wherein the trained machine-learning model comprises a neural network.

12. The method of claim 1 , further comprising:

providing the output data to a text-to-speech system for producing speech output based on the output data.

13. The method of claim 1 , wherein the text data includes a text-based word having the at least one character that is the non-letter character, wherein the non-letter character appears in a location in the text-based word normally occupied by a letter.

14. The method of claim 1 , further comprising storing the second portion as an alias for the associated portion in an alias database, the alias database having one or more other aliases.

15. A system comprising memory storing instructions that, when executed by one or more processors, cause the one or more processors to:

receive text data including at least one character that is a non-letter character, the text data including at least one word that is spelled differently than a standard spelling;

provide the text data as input into a trained machine-learning model; and

receive as output from the trained machine-learning model, output data indicative of a pronunciation of the text data, the pronunciation corresponding to the standard spelling of the at least one word;

wherein the trained machine learning model is trained using an utterance from a user, wherein the training includes instructions that, when executed by the one or more processors, cause the one or more processors to:

process the utterance with a speech-to-text engine to identify within the utterance at least a first and a second portion;

conduct a search to find a match to the first portion in a metadata database, the match having associated data, wherein the metadata database includes metadata which describes media content, and the metadata database includes at least the following categories of the metadata: song titles, artist names, and albums, wherein the match and the associated data are each associated with different categories of the metadata in the metadata database;

analyze the associated data to identify an associated portion in the associated data that has a similarity to the second portion, wherein the associated portion includes a non-letter character; and

train the machine-learning model with the associated portion as training input data and the second portion as training output data.

16. The system of claim 15 , further comprising media streaming application instructions stored in a non-transitory memory of a voice-interactive device executable to cause operation of a media streaming application on the voice-interactive device.

17. The system of claim 15 , wherein training data includes text-based words and associated correct pronunciations.

18. The system of claim 17 , wherein training data includes at least a portion of the text-based word identified as having a correct pronunciation different than a phonetic pronunciation.

19. The system of claim 17 , wherein training the machine-learning module includes training the machine-learning module to produce pronunciation output based on text input.

20. A computer-implemented method comprising:

receiving an utterance from a user;

obtaining text data from the utterance;

providing the text data as input into a trained machine-learning model; and

receiving as output from the trained machine-learning model, output data including alternative text data based on the utterance, the alternative text data including at least one character that is a non-letter character, the alternative text data being associated with a pronunciation of the utterance; and

wherein the trained machine learning model is trained using the utterance, wherein the training includes:

processing the utterance with a speech-to-text engine to identify within the utterance at least a first and a second portion;

conducting a search to find a match to the first portion in a metadata database, the match having associated data, wherein the metadata database includes metadata which describes media content, and the metadata database includes at least the following categories of the metadata: song titles, artist names, and albums, wherein the match and the associated data are each associated with different categories of the metadata in the metadata database;

analyzing the associated data to identify an associated portion in the associated data that has a similarity to the second portion, wherein the associated portion includes a non-letter character; and

training the machine-learning model with the associated portion as training input data and the second portion as training output data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 17, 2019
From: BROMAND, DANIEL
To: SPOTIFY AB
Reel/Frame 050748/0632 →
Continuity (1)
Related Publication 20200357390A1 · Nov 12, 2020
Cited By (1)
US 12,525,232