IP Library Granted Patent US 11,594,221
Granted Patent B2
US 11,594,221 · App. 17/212,846 · Granted Feb 28, 2023

Transcription generation from multiple speech recognition systems

Inventors: David Thomson (North Salt Lake, UT); Jadie Adams (Salt Lake City, UT); Jonathan Skaggs (Provo, UT); Joshua McClellan (Silver Spring, MD); Shane Roylance (Farmington, UT)
Assignee: Sorenson IP Holdings, LLC
G10L15/22G10L15/187G10L15/26G10L15/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,594,221
App. No.
17/212,846
Granted
Feb 28, 2023
Kind
B2
Abstract

A method may include obtaining first audio data originating at a first device during a communication session between the first device and a second device. The method may also include obtaining a first text string that is a transcription of the first audio data, where the first text string may be generated using automatic speech recognition technology using the first audio data. The method may also include obtaining a second text string that is a transcription of second audio data, where the second audio data may include a revoicing of the first audio data by a captioning assistant and the second text string may be generated by the automatic speech recognition technology using the second audio data. The method may further include generating an output text string from the first text string and the second text string and using the output text string as a transcription of the speech.

Claims (52)

1. A method comprising:

obtaining first audio data that includes speech;

obtaining a first text string that is a transcription of the first audio data, the first text string generated by speech recognition technology using the first audio data;

obtaining a second text string that is a transcription of second audio data, the second audio data including a revoicing of the first audio data and the second text string generated by the speech recognition technology using the second audio data;

obtaining a third text string that is a transcription of the first audio data or the second audio data, the third text string generated using the speech recognition technology; and

generating an output text string from the first text string, the second text string, and the third text string, wherein the output text string includes one or more first words from one of the first text string, the second text string, and the third text string and one or more second words from another one of the first text string, the second text string, and the third text string such that the output text string, does not include an entirety of any of the first text string, the second text string, and the third text string.

2. The method of claim 1 , further comprising providing the output text string as a transcription of the speech.

3. The method of claim 1 , wherein the speech recognition technology used to generate the first text string includes a first model trained for a plurality of individuals and the speech recognition technology used to generate the second text string includes a second model adapted to a captioning assistant performing the revoicing.

4. The method of claim 1 , wherein the output text string includes one or more first words from the first text string and one or more second words from the second text string.

5. The method of claim 1 , further comprising correcting at least one word in one or more of: the output text string, the first text string, and the second text string based on input obtained from a device associated with the revoicing.

6. The method of claim 1 , wherein the first text string, the second text string, and the third text string are hypothesis generated by the speech recognition technology for the same portion of the first audio data.

7. At least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by at least one computing system cause performance of the method of claim 1 .

8. A method comprising:

obtaining first audio data that includes speech;

obtaining a first text string based on the first audio data, the first text string generated by speech recognition technology using the first audio data;

obtaining a second text string based on the first audio data, the second text string generated by the speech recognition technology using the first audio data;

obtaining a third text string based on the first audio data, the third text string generated by the speech recognition technology using the first audio data; and

generating an output text string from the first text string, the second text string, and the third text string, wherein the output text string includes one or more first words from one of the first text string, the second text string, and the third text string and one or more second words from another one of the first text string, the second text string, and the third text string such that the output text string does not include an entirety of any of the first text string, the second text string, and the third text string.

9. The method of claim 8 , wherein the generation of the second text string includes the speech recognition technology using a revoicing of the first audio data to generate the second text string.

10. The method of claim 8 , wherein the first text string is generated using a first speech recognition model, the second text string is generated using a second speech recognition model different from the first speech recognition model, and the third text string is generated using a third speech recognition model different from the first speech recognition model and the second speech recognition model.

11. The method of claim 8 , wherein the first text string and the third text string are both hypotheses generated by the speech recognition technology for the same portion of the first audio data.

12. The method of claim 8 , wherein the first text string is a hypotheses generated by a first speech recognition model and the second text string is a hypotheses generated by a second speech recognition model for the same portion of the first audio data.

13. At least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by at least one computing system cause performance of the method of claim 8 .

14. A method comprising:

obtaining first audio data that includes speech;

obtaining a first text string that is a transcription of the first audio data, the first text string generated by speech recognition technology using the first audio data;

obtaining a second text string that is a transcription of second audio data, the second audio data including a revoicing of the first audio data and the second text string generated by the speech recognition technology using the second audio data; and

generating an output text string from the first text string and the second text string, the output text string includes one or more first words from the first text string and one or more second words from the second text string such that the output text string does not include an entirety of the first text string or an entirety of the second text string.

15. The method of claim 14 , further comprising providing the output text string for presentation without providing the first text string and the second text string for presentation.

16. The method of claim 14 , wherein generating the output text string includes:

selecting the one or more second words based on the first text string and the second text string both including the one or more second words; and

selecting the one or more first words from the first text string based on the second text string not including the one or more first words.

17. The method of claim 14 , wherein generating the output text string includes:

selecting the one or more first words based on the first text string and the second text string both including the one or more first words; and

selecting the one or more second words from the second text string based on the first text string not including the one or more second words.

18. The method of claim 14 , further comprising correcting at least one word in one or more of: the output text string, the first text string, and the second text string based on a third text string generated by the speech recognition technology using the first audio data.

19. The method of claim 18 , wherein the first text string and the third text string are both hypotheses generated by the speech recognition technology for the same portion of the first audio data.

20. At least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by at least one computing system cause performance of the method of claim 14 .

21. A method comprising:

obtaining first audio data that includes speech;

obtaining a first text string based on the first audio data, the first text string generated by speech recognition technology using the first audio data;

obtaining a second text string based on the first audio data, the second text string generated by speech recognition technology using the first audio data; and

generating an output text string using the first text string in response to the first text string being generated and without regard to a quality measure of the first text string, wherein in response to the second text string having the quality measure satisfying the quality threshold, the output text string is generated using the first text string and the second text string instead of generating the output text string using only the first text string.

22. The method of claim 21 , further comprising correcting at least one word in one or more of: the output text string, the first text string, and the second text string based on a third text string generated by the speech recognition technology using the first audio data.

23. The method of claim 22 , wherein the first text string and the third text string are both hypotheses generated by the speech recognition technology for the same portion of the first audio data.

24. At least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by at least one computing system cause performance of the method of claim 21 .

25. A method comprising:

obtaining first audio data that includes speech;

obtaining a first text string that is a transcription of the first audio data, the first text string generated by speech recognition technology using the first audio data;

obtaining a second text string that is a transcription of second audio data, the second audio data including a revoicing of the first audio data and the second text string generated by the speech recognition technology using the second audio data; and

in response to one of the first text string and the second text string having a quality measure satisfying a quality threshold, generating an output text string using the first text string and the second text string instead of generating the output text string using only another of the first text string and the second text string in response to the quality measure of the one of the first text string and the second text string not satisfying the quality threshold.

26. At least one non-transitory computer-readable media configured to store one or more instructions that in response to being executed by at least one computing system cause performance of the method of claim 25 .

Assignments (6)
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA THE NAME OF THE LAST RECEIVING PARTY SHOULD BE CAPTIONCALL, LLC PREVIOUSLY RECORDED ON REEL 67190 FRAME 517. ASSIGNOR(S) HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST. Recorded May 31, 2024
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
To: SORENSON IP HOLDINGS, LLC; SORENSON COMMUNICATIONS, LLC; CAPTIONCALL, LLC
Reel/Frame 067591/0675 →
RELEASE OF SECURITY INTEREST Recorded Apr 23, 2024
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
To: SORENSON IP HOLDINGS, LLC; SORENSON COMMUNICATIONS, LLC; CAPTIONALCALL, LLC
Reel/Frame 067190/0517 →
SECURITY INTEREST Recorded Apr 23, 2024
From: SORENSON COMMUNICATIONS, LLC; INTERACTIVECARE, LLC; CAPTIONCALL, LLC
To: OAKTREE FUND ADMINISTRATION, LLC, AS COLLATERAL AGENT
Reel/Frame 067573/0201 →
JOINDER NO. 1 TO THE FIRST LIEN PATENT SECURITY AGREEMENT Recorded Apr 22, 2021
From: SORENSON IP HOLDINGS, LLC
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
Reel/Frame 056019/0204 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 25, 2021
From: HOLM, MICHAEL; BLACK, DAVID; BAROCIO, JESSE; THOMSON, DAVID; BOEKWEG, SCOTT; ROYLANCE, SHANE; CLEMENTS, KIERSTEN; BOEHME, KENNETH; ADAMS, JADIE; SKAGGS, JONATHAN; ORZECHOWSKI, GRZEGORZ; MCCLELLAN, JOSHUA
To: CAPTIONCALL, LLC
Reel/Frame 055723/0388 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 25, 2021
From: CAPTIONCALL, LLC
To: SORENSON IP HOLDINGS, LLC
Reel/Frame 055723/0416 →