IP Library Granted Patent US 11,862,152
Granted Patent B2
US 11,862,152 · App. 17/214,462 · Granted Jan 2, 2024

Dynamic domain-adapted automatic speech recognition system

Inventors: Atul Kumar (San Jose, CA); Elizabeth O. Bratt (Mountain View, CA); Minsuk Heo (Santa Clara, CA); Nidhi Rajshree (Los Gatos, CA); Praful Chandra Mangalath (Sunnyvale, CA)
Assignee: ROKU, INC.
G10L15/187G06F40/205G06F40/295G10L15/1822G10L15/22G10L25/33
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,862,152
App. No.
17/214,462
Granted
Jan 2, 2024
Kind
B2
Abstract

Disclosed herein are system, apparatus, article of manufacture, method, and computer program product embodiments for adapting an automated speech recognition system to provide more accurate suggestions to voice queries involving media content including recently created or recently available content. An example computer-implemented method includes transcribing the voice query, identifying respective components of the query such as the media content being requested and the action to be performed, and generating fuzzy candidates that potentially match the media content based on phonetic representations of the identified components. Phonetic representations of domain specific candidates are stored in a domain entities index and is continuously updated with new entries so as to maintain the accuracy of the speech recognition of voice queries for recently created or recently available content.

Claims (71)

1. A computer-implemented method, by a display device, for adapting an automatic speech recognition engine, comprising:

receiving a voice query that includes an action and requested media content;

generating a transcription of the voice query, wherein the transcription is generated using the automatic speech recognition engine and wherein the transcription includes a textual representation of the requested media content;

parsing the transcription to identify an entity corresponding to the textual representation of the media content;

generating a phonetic representation of the entity, wherein the phonetic representation includes at least one of a grapheme of the entity, a phoneme of the entity, and an N-gram of the entity;

generating, based on the phonetic representation, a fuzzy candidate list comprising a plurality of fuzzy candidates representing potential matches to the requested media content, wherein generating the fuzzy candidate list comprises:

utilizing a lossy phonetic form of the phonetic representation to generate a predetermined number of candidates; and

utilizing a precise phonetic form of the phonetic representation to generate the fuzzy candidate list by reducing the predetermined number of candidates;

ranking the fuzzy candidate list to form a ranked fuzzy candidate list including a highest ranked fuzzy candidate corresponding to a best potential match for the requested media content;

displaying, on the display device, the ranked fuzzy candidate list;

receiving, from a remote control in communication with the display device, user input for a selected fuzzy candidate from the ranked fuzzy candidate list; and

performing the action on the selected fuzzy candidate.

2. The computer-implemented method of claim 1 , wherein the highest ranked fuzzy candidate corresponding to the best potential match for the requested media content is determined based on a ranking criteria including at least one of a phonetic edit distance, a popularity score, a match count, a longest common sequence score, and a nospace overlap score.

3. The computer-implemented method of claim 1 , wherein the action includes at least one of receiving a selection of the highest ranked fuzzy candidate from the display device, retrieving the highest ranked fuzzy candidate from a database, and sending the ranked fuzzy list including the highest ranked fuzzy candidate to the display device.

4. The computer-implemented method of claim 1 , further comprising:

receiving, from an entertainment domain entity source, a second media content;

converting the second media content into a second phoneme, wherein the second phoneme is a phonetic representation of the second media content; and

storing the second media content and the second phoneme as an entry in a domain entity index.

5. The computer-implemented method of claim 1 , further comprising:

determining an intent of the voice query based on the action and the requested media content.

6. The computer-implemented method of claim 5 , wherein the intent of the voice query is a content request and the action is a command to play the requested media content.

7. The computer-implemented method of claim 1 , wherein the generating further comprises:

performing a grapheme search based on the grapheme of the entity to identify at least one fuzzy grapheme candidate based on a spelling comparison between the grapheme of the entity and the at least one fuzzy grapheme candidate, wherein the plurality of fuzzy candidates comprises the at least one fuzzy grapheme candidate.

8. The computer-implemented method of claim 7 , wherein the spelling comparison comprises:

using the grapheme of the entity to search for a grapheme candidate in a domain entity index; and

identifying the grapheme candidate as the at least one fuzzy grapheme candidate based on matching a spelling of the grapheme to a spelling of the grapheme candidate.

9. The computer-implemented method of claim 8 , wherein the domain entity index comprises an entry associated with the grapheme candidate, the computer-implemented method further comprising:

populating the entry with the spelling of the grapheme candidate independently of the automatic speech recognition engine; and

retrieving, from the entry, the spelling of the grapheme candidate.

10. The computer-implemented method of claim 9 , wherein the domain entity index comprises a plurality of entries, including the entry, associated with a plurality of grapheme candidates and wherein the domain entity index is updated on a continuous basis.

11. The computer-implemented method of claim 7 , wherein the generating further comprises:

performing a phoneme search based on the phoneme of the entity to identify at least one fuzzy phoneme match based on a phonetic comparison between the phoneme of the entity and the at least one fuzzy phoneme candidate, wherein the plurality of fuzzy candidates further comprises the at least one fuzzy phoneme candidate.

12. The computer-implemented method of claim 11 , wherein the phonetic comparison comprises:

using the phoneme of the entity to search for a phoneme candidate in a domain entity index; and

identifying the phoneme candidate as the at least one fuzzy phone candidate based on a phonetic matching between the phoneme of the entity and the phoneme candidate.

13. The computer-implemented method of claim 12 , wherein the domain entity index comprises an entry associated with the phoneme candidate, the computer-implemented method further comprising:

populating the entry with the phoneme candidate independently of the automatic speech recognition engine; and

retrieving, from the entry, the phoneme candidate.

14. The computer-implemented method of claim 11 , wherein the generating further comprises:

performing an N-gram search based on the N-gram of the entity to identify at least one fuzzy N-gram match based on an N-gram comparison between the entity and the at least one fuzzy N-gram candidate, wherein the plurality of fuzzy candidates further comprises the at least one fuzzy N-gram match.

15. The computer-implemented method of claim 14 , wherein the ranking further comprises:

ranking the at least one fuzzy grapheme match, the at least one fuzzy N-gram match, the at least one fuzzy phoneme match in the fuzzy candidate list to form the ranked candidate list.

16. The computer-implemented method of claim 14 , wherein the N-gram comparison comprises:

using the N-gram of the entity to search for an N-gram candidate in a domain entity index; and

identifying the N-gram candidate as the at least one fuzzy N-gram candidate based on matching the N-gram of the entity to an N-gram of the N-gram candidate.

17. The computer-implemented method of claim 16 , wherein the domain entity index comprises an entry associated with the phoneme candidate, the computer-implemented method further comprising:

retrieving, from the entry, the N-gram of the N-gram candidate.

18. An apparatus comprising:

a memory; and

a processor communicatively coupled to the memory and configured to:

receive a voice query including an action and requested media content;

generate a transcription of the voice query, wherein the transcription is generated using an automatic speech recognition engine and wherein the transcription includes a textual representation of the requested media content and wherein the textual representation is an imperfect match to the requested media content;

generate a phonetic representation of the textual representation of the requested media content;

generate, based on the phonetic representation, a fuzzy candidate list comprising a plurality of fuzzy candidates representing potential matches to the requested media content, wherein generating the fuzzy candidate list comprises:

utilizing a lossy phonetic form of the phonetic representation to generate a predetermined number of candidates; and

utilizing a precise phonetic form of the phonetic representation to generate the fuzzy candidate list by reducing the predetermined number of candidates;

rank the fuzzy candidate list to form a ranked fuzzy candidate list including a highest ranked fuzzy candidate corresponding to a best potential match for the requested media content;

display, on the apparatus, the ranked fuzzy candidate list;

receive, from a remote control in communication with the apparatus, user input for a selected fuzzy candidate from the ranked fuzzy candidate list; and

perform the action on the selected fuzzy candidate.

19. A non-transitory computer-readable medium storing instructions, wherein the instructions, when executed by a processor of a display device, cause the processor to perform operations comprising:

receiving a voice query including an action and requested media content;

generating a transcription of the voice query, wherein the transcription is generated using an automatic speech recognition engine and wherein the transcription includes a textual representation of the requested media content and wherein the textual representation is an imperfect match to the requested media content;

generating a phonetic representation of the textual representation of the requested media content;

generating, based on the phonetic representation, a fuzzy candidate list comprising a plurality of fuzzy candidates representing potential matches to the requested media content, wherein generating the fuzzy candidate list comprises:

utilizing a lossy phonetic form of the phonetic representation to generate a predetermined number of candidates; and

utilizing a precise phonetic form of the phonetic representation to generate the fuzzy candidate list by reducing the predetermined number of candidates;

ranking the fuzzy candidate list to form a ranked fuzzy candidate list including a highest ranked fuzzy candidate corresponding to a best potential match for the requested media content;

displaying, on the display device, the ranked fuzzy candidate list;

receiving, from a remote control in communication with the display device, user input for a selected fuzzy candidate from the ranked fuzzy candidate list; and

performing the action on the highest ranked selected fuzzy candidate.

Assignments (2)
SECURITY INTEREST Recorded Sep 18, 2024
From: ROKU, INC.
To: CITIBANK, N.A.
Reel/Frame 068982/0377 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 12, 2021
From: HEO, MINSUK; BRATT, ELIZABETH O.; MANGALATH, PRAFUL CHANDRA; KUMAR, ATUL; RAJSHREE, NIDHI
To: ROKU, INC.
Reel/Frame 058096/0583 →
Continuity (1)
Related Publication 20220310076A1 · Sep 29, 2022
Cited By (1)
US 12,374,328