Granted Patent B1

US 10,573,312 · App. 16/209,623 · Granted Feb 25, 2020

Inventors: David Thomson (North Salt Lake, UT); Jadie Adams (Salt Lake City, UT); Jonathan Skaggs (Provo, UT); Joshua McClellan (Silver Spring, MD); Shane Roylance (Farmington, UT)

Assignee: Sorenson IP Holdings, LLC

G10L15/22G10L15/187G10L15/30

Abstract

Claims (57)

1. A method comprising:

obtaining first audio data originating at a first device during a communication session between the first device and a second device, the communication session configured for verbal communication such that the first audio data includes speech;

obtaining a first text string that is a transcription of the first audio data, the first text string generated by a first automatic speech recognition system using the first audio data and using a first model trained for a plurality of individuals;

obtaining a second text string that is a transcription of second audio data, the second audio data including a revoicing of the first audio data by a captioning assistant and the second text string generated by a second automatic speech recognition system using the second audio data and using a second model trained for the captioning assistant;

obtaining a third text string that is a transcription of the first audio data or the second audio data, the third text string generated by a third automatic speech recognition system using a third model;

generating an output text string from the first text string, the second text string, and the third text string; and

providing the output text string as a transcription of the speech to the second device for presentation during the communication session concurrently with the presentation of the first audio data by the second device.

2. The method of claim 1 , wherein the first model includes one or more of the following: a feature model, a transform model, an acoustic model, a language model, and a pronunciation model.

3. The method of claim 1 , wherein generating the output text string further includes:

de-normalizing the first text string and the second text string;

aligning the first text string and the second text string; and

comparing the aligned and de-normalized first and second text strings.

4. The method of claim 1 , wherein generating the output text string further includes:

selecting one or more second words from the second text string for the output text string based on the first text string and the second text string both including the one or more second words; and

selecting one or more first words from the first text string for the output text string based on the second text string not including the one or more first words.

5. The method of claim 1 , further comprising correcting at least one word in one or more of: the output text string, the first text string, and the second text string based on input obtained from a device associated with the captioning assistant.

6. The method of claim 5 , wherein the input obtained from the device is based on a fourth text string generated by the first automatic speech recognition system using the first audio data.

7. The method of claim 6 , wherein the first text string and the fourth text string are both hypothesis generated by the first automatic speech recognition system for the substantially same portion of the first audio data.

8. The method of claim 1 , wherein the third text string is a transcription of the first audio data, the method further comprising obtaining a fourth text string that is a transcription of the second audio data, the fourth text string generated by a fourth automatic speech recognition system using the second audio data and using a fourth model, wherein the output text string is generated from the first text string, the second text string, the third text string, and the fourth text string.

9. The method of claim 1 , further comprising:

obtaining fourth audio data that includes speech and that originates at the first device during the communication session;

obtaining a third text string that is a transcription of the fourth audio data, the fourth text string generated by the first automatic speech recognition system using the fourth audio data and using the first model; and

in response to either no revoicing of the fourth audio data or a fifth transcription generated using the second automatic speech recognition system having a quality measure satisfying a quality threshold, generating a second output text string using only the fourth text string.

10. At least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by at least one computing system cause performance of the method of claim 1 .

11. A method comprising:

obtaining first audio data originating at a first device during a communication session between the first device and a second device, the communication session configured for verbal communication such that the first audio data includes speech;

obtaining a first text string that is a transcription of the first audio data, the first text string generated using automatic speech recognition technology using the first audio data;

obtaining a second text string that is a transcription of second audio data, the second audio data including a revoicing of the first audio data by a captioning assistant and the second text string generated by the automatic speech recognition technology using the second audio data;

obtaining a third text string that is a transcription of the first audio data or the second audio data, the third text string generated by the automatic speech recognition technology;

generating an output text string from the first text string, the second text string, and the third text string; and

using the output text string as a transcription of the speech.

12. The method of claim 11 , wherein the automatic speech recognition technology used to generate the first text string is a first automatic speech recognition system that includes a first model trained for a plurality of individuals and the automatic speech recognition technology used to generate the second text string is a second automatic speech recognition system that includes a second model adapted to the captioning assistant.

13. The method of claim 11 , wherein the output text string includes one or more first words from the first text string and one or more second words from the second text string.

14. The method of claim 11 , further comprising correcting at least one word in one or more of: the output text string, the first text string, and the second text string based on input obtained from a device associated with the captioning assistant.

15. The method of claim 14 , wherein the input obtained from the device is based on a fourth text string generated by the automatic speech recognition technology using the first audio data.

16. The method of claim 15 , wherein the first text string and the fourth text string are both hypothesis generated by the automatic speech recognition technology for the substantially same portion of the first audio data.

17. The method of claim 11 , further comprising:

obtaining third audio data that includes speech and that originates at the first device during the communication session;

obtaining a fourth text string that is a transcription of the third audio data, the fourth text string generated by the automatic speech recognition technology using the third audio data; and

in response to either no revoicing of the third audio data or a fourth transcription, generated using the automatic speech recognition technology and revoicing of the third audio data, having a quality measure satisfying a quality threshold, generating a second output text string using only the fourth text string.

18. At least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by at least one computing system cause performance of the method of claim 11 .

19. A method comprising:

obtaining first audio data originating at a first device during a communication session between the first device and a second device, the communication session configured for verbal communication such that the first audio data includes speech;

obtaining a first text string that is a transcription of the first audio data, the first text string generated using automatic speech recognition technology using the first audio data;

obtaining a second text string that is a transcription of second audio data, the second audio data including a revoicing of the first audio data and the second text string generated by the automatic speech recognition technology using the second audio data;

obtaining a third text string that is a transcription of the first audio data or the second audio data, the third text string generated by the automatic speech recognition technology;

generating an output text string from the first text string, the second text string, and the third text string, the output text string including one or more words based on at least two of the first text string, the second text string, and the third text string including the one or more words; and

providing the output text string as a transcription of the speech to the second device for presentation during the communication session by the second device.

20. A system comprising:

one or more processors; and

at least one non-transitory computer-readable media coupled to the one or more processors, the at least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by the one or more processors cause the system to perform operations, the operations comprising:

obtain first audio data originating at a first device during a communication session between the first device and a second device, the communication session configured for verbal communication such that the first audio data includes speech;

obtain a first text string that is a transcription of the first audio data, the first text string generated using automatic speech recognition technology using the first audio data;

obtain a second text string that is a transcription of second audio data, the second audio data including a revoicing of the first audio data and the second text string generated by the automatic speech recognition technology using the second audio data;

obtain a third text string that is a transcription of the first audio data or the second audio data, the third text string generated by the automatic speech recognition technology;

generate an output text string from the first text string, the second text string, and the third text string; and

provide the output text string as a transcription of the speech.

Assignments (11)

Cited By (82)

US 50,388 US 50,834 US 12,190,870 US 12,211,490 US 12,212,945 US 12,217,748 US 12,217,758 US 12,217,765 US 12,230,291 US 12,236,932 US 12,236,955 US 12,248,748 US 12,249,313 US 12,271,695 US 12,278,925 US 12,283,269 US 12,284,059 US 12,288,558 US 12,306,809 US 12,308,030 US 12,314,263 US 12,314,633 US 12,314,672 US 12,321,853 US 12,322,390 US 12,323,554 US 12,327,549 US 12,327,556 US 12,334,054 US 12,341,621 US 12,360,734 US 12,373,027 US 12,375,052 US 12,380,896 US 12,387,716 US 12,400,661 US 12,406,672 US 12,406,684 US 12,412,583 US 12,424,220 US 12,430,569 US 12,436,994 US 12,438,977 US 12,456,465 US 12,462,802 US 12,462,808 US 12,468,883 US 12,469,491 US 12,475,883 US 12,481,841 US 12,488,785 US 12,494,929 US 12,498,899 US 12,505,832 US 12,511,098 US 12,511,498 US 12,513,479 US 12,518,748 US 12,518,755 US 12,518,756 US 12,525,242 US 12,530,111 US 12,530,112 US 12,555,581 US 12,561,536 US 12,562,152 US 12,567,416 US 12,567,418 US 12,579,978 US 12,579,982 US 12,608,542 US 12,619,818 US 12,626,717 US 12,645,874 US 12,652,260 US 12,658,185 US 12,670,911 US 12,675,484 US 12,682,187 US 12,699,543 US 12,706,092 US 12,711,962