SEMIAUTOMATED RELAY METHOD AND APPARATUS
A method and system for providing captioned telephone service, the method comprising the steps of, initiating a first captioned telephone service call, during the first captioned telephone service call, creating a first set of captions using a call assistant, simultaneous with creating the first set of captions using a call assistant, creating a second set of captions using an automated speech recognition engine, comparing the first set of captions and the second set of captions using a scoring algorithm based on errors between the first and second sets of captions to generate a score for the second set of captions, in response to the score being within a predetermined threshold range, continuing the call using only the automated speech recognition engine to generate text and in response to the score being outside of the predetermined threshold range, continuing the call using a call assistant to generate captions.
1 . A caption device for use by a hard of hearing assisted user (AU) to assist the AU during voice communications with a hearing user (HU) using an HU device, the caption device comprising:
a display screen;
a memory;
at least one communication link element for linking to a communication network;
a speaker;
a captioning actuator input feature for receiving a request to initiate captioning of an HU voice signal during a voice communication session between the caption device and the HU device;
at least one processor programmed to perform the steps of:
(i) receiving an HU voice signal from the HU device during a call;
(ii) storing at least a most recent portion of the HU voice signal in the memory;
(iii) broadcasting the HU voice signal to the AU via the speaker;
(iv) receiving, prior to any captioning of the most recent portion of the HU voice signal that is stored in memory, a caption request via the captioning actuator;
(v) upon receiving the caption request, retrieving the most recent portion of the HU voice signal from the memory;
(vi) obtaining a text caption corresponding to the most recent portion of the HU voice signal; and
(vi) presenting the text caption to the AU via the display screen.
2 . The device of claim 1 wherein the step of obtaining a text caption includes initiating a process whereby an automated speech recognition (ASR) engine converts the retrieved HU voice signal to text.
3 . The device of claim 2 wherein the ASR is run by the at least one processor.
4 . The device of claim 2 wherein, after presenting the text caption via the display screen, the ASR engine error corrects the text, the at least one processor using the corrections to make in line corrections to the text captions presented to the AU via the display screen.
5 . The device of claim 4 wherein the in-line corrections are visually distinguished from other text that is not corrected.
6 . The device of claim 1 for use with a remote relay that performs captioning services, the step of obtaining a text caption including the at least one processor transmitting the retrieved most recent portion of the HU voice signal to the relay and receiving the text caption back from the relay.
7 . The device of claim 6 further including, after receiving the caption request, establishing a communication link with the relay.
8 . The device of claim 1 wherein the at least one processor is further programmed to continue obtaining text captions associated with HU voice signal received during the call for at least a period after receiving the caption request and presenting the text captions associated with the HU voice signal received subsequent to the request via the display screen.
9 . The device of claim 3 wherein the at least one processor is further programmed to continue obtaining text captions associated with HU voice signal received during the call for at least a period after receiving the caption request and presenting the text captions associated with the HU voice signal received subsequent to the request via the display screen.
10 . The device of claim 9 for use with a remote relay that performs captioning services, the step of obtaining text captions for voice signal received after the caption request including the at least one processor transmitting at least a portion of the HU voice signal received after the caption request to the relay and receiving the text caption back from the relay.
11 . The device of claim 10 wherein at least a portion of the HU voice signal received after the caption request is also transcribed by the ASR and the resulting text caption is presented via the display screen.
12 . The device of claim 1 wherein the step of obtaining a text caption includes performing at least first and second different transcription processes to generate first and second text captions corresponding to the retrieved HU voice signal and using the first and second text captions to derive an output text caption that is presented via the display screen.
13 . The device of claim 12 wherein the first transcription process includes using an automatic speech recognition (ASR) engine to generate the first text caption and the second transcription process includes a re-voicing process whereby a call assistant listens to the retrieved voice signal and revoices the retrieved voice signal to an ASR to generate the second text caption.
14 . A caption device for use by a hard of hearing assisted user (AU) to assist the AU during voice communications with a hearing user (HU) using an HU device, the caption device comprising:
a display screen;
a memory;
at least one communication link element for linking to a communication network;
a speaker;
a captioning actuator input feature for receiving a request to initiate captioning of an HU voice signal during a voice communication session between the caption device and the HU device;
at least one processor programmed to perform the steps of:
(i) receiving an HU voice signal from the HU device during a call;
(ii) storing at least a most recent portion of the HU voice signal in the memory;
(iii) broadcasting the HU voice signal to the AU via the speaker;
(iv) receiving, prior to any captioning of the most recent portion of the HU voice signal that is stored in memory, a caption request via the captioning actuator;
(v) upon receiving the caption request, retrieving the most recent portion of the HU voice signal from the memory;
(vi) using an automatic speech recognition (ASR) program to generate a text caption corresponding to the most recent portion of the HU voice signal; and
(vi) presenting the text caption to the AU via the display screen.
15 . The device of claim 14 wherein, after presenting the text caption via the display screen, the ASR engine error corrects the text, the at least one processor using the corrections to make in line corrections to the text captions presented to the AU via the display screen.
16 . The device of claim 15 wherein the in-line corrections are visually distinguished from other text that is not corrected.
17 . The device of claim 12 wherein the at least one processor is further programmed to continue generating text captions associated with HU voice signal received during the call for at least a period after receiving the caption request and presenting the text captions associated with the HU voice signal received subsequent to the request via the display screen.
18 . The device of claim 17 wherein text captions corresponding to the retrieved voice signal are visually distinguished from text captions corresponding to HU voice signal received after the caption request is received.
19 . The device of claim 12 for use with a remote relay that performs caption error correction services, the at least one processor transmitting the generated captions and the HU voice signal to the relay as it is generated and receiving error corrections to the generated captions back from the relay and using the error corrections to make in line corrections to captions presented via the display screen.
20 . A caption device for use by a hard of hearing assisted user (AU) to assist the AU during voice communications with a hearing user (HU) using an HU device, the caption device comprising:
a display screen;
a memory;
at least one communication link element for linking to a communication network;
a speaker;
a captioning actuator input feature for receiving a request to initiate captioning of an HU voice signal during a voice communication session between the caption device and the HU device;
at least one processor programmed to perform the steps of:
(i) receiving an HU voice signal from the HU device during a call;
(ii) storing at least a most recent portion of the HU voice signal in the memory;
(iii) broadcasting the HU voice signal to the AU via the speaker;
(iv) receiving, prior to any captioning of the most recent portion of the HU voice signal that is stored in memory, a caption request via the captioning actuator;
(v) upon receiving the caption request, retrieving the most recent portion of the HU voice signal from the memory;
(vi) running an automatic speech recognition (ASR) program to generate a text caption corresponding to the most recent portion of the HU voice signal;
(vi) presenting the text caption to the AU via the display screen;
(vii) after receiving the caption request:
(a) continuing to receive the HU voice signal during the ongoing call for at least a duration of time;
(b) obtaining text captions corresponding to the continuing voice signal; and
(c) presenting the text captions corresponding to the continuing voice signal via the display screen.