Automatic translation between sign language and spoken language
Methods, apparatus, systems, and articles of manufacture to translation between sign language and spoken language are disclosed. An example apparatus includes processor circuitry to at least one of instantiate or execute machine readable instructions to identify a plurality of candidate signs across different frames in video; associate a respective gloss to respective ones of the candidate signs; associate a respective confidence score with the respective glosses; identify overlapping frames of the candidate signs; select one or more of the candidate signs as performed signs based on the respective confidence scores and overlapping frames; and convert the performed signs to audio data.
1 . An apparatus to translate between sign language and spoken language, the apparatus comprising:
at least one memory;
machine readable instructions stored in the memory; and
processor circuitry to at least one of instantiate or execute the machine readable instructions to perform operations to translate a real-time conversation between sign language and spoken language, the operations including:
obtain a continuous sign video that captures signs generated during the real-time conversation;
create a list of a plurality of random video segments of the continuous sign video, the plurality of video segments including random start and end times during a duration of the continuous sign video;
identify, via execution of a trained single sign language recognition model that is provided the continuous sign video and the list of random video segments, a plurality of candidate signs, each of the plurality of candidate signs identified based on features identified in respective subsets of the plurality of video segments, wherein at least two of the subsets of the plurality of video segments include one or more of the same video segments;
associate, via the execution of the trained single sign language recognition model, a respective gloss to each of the plurality of candidate signs;
generate, via the execution of the trained single sign language recognition model, a respective confidence score for each of the respective glosses, the confidence score indicating a probability identified by the trained single sign language recognition model that the respective glosses are an accurate translation of the plurality of candidate signs;
determine which of the plurality of candidate signs are identified using one or more of the same video segments;
select one or more of the plurality of candidate signs as performed signs based on the respective confidence scores and which of the plurality of candidate signs are identified using one or more of the same video segments; and
generate, via a model, audio data that represents the performed signs, the audio data stored in the memory for conversion to sound waves for the conversation.
2 . The apparatus of claim 1 , wherein to generate the audio data, the processor circuitry is to convert the performed signs to text.
3 . The apparatus of claim 1 , wherein the processor circuitry is to transmit the audio data to a user device including a speaker for conversion of the audio data to sound.
4 . The apparatus of claim 1 , wherein the processor circuitry is to:
identify non-manual elements in the continuous sign video; and
adjust the audio data, by adjusting at least one of a volume, cadence, tone, or emotion of speech in the audio data based on the non-manual elements.
5 . The apparatus of claim 4 , wherein the non-manual elements include at least one of a facial expression, an eye gaze, a rhythm, a cadence, or a pause.
6 . The apparatus of claim 1 , wherein the processor circuitry is to:
identify frames of the continuous sign video without sign language as non-useful frames; and
remove the non-useful frames of the continuous sign video prior to identifying the plurality of candidate signs.
7 . The apparatus of claim 1 , wherein the processor circuitry is to:
identify a movement of a person in the video as a non-sign language movement; and
disregard the non-sign language movement prior to identifying the plurality of candidate signs.
8 . The apparatus of claim 1 , wherein the processor circuitry is to:
identify respective time stamps of the plurality of video segments associated with the performed signs;
order the performed signs based on the time stamps; and
convert the performed signs to the audio data based on the order.
9 . The apparatus of claim 1 , wherein the processor circuitry is to:
detect a speed of signing in the video;
perform a comparison of the speed to a threshold speed; and
change a number of video frames in the plurality of video segments analyzed to identify the plurality of candidate signs based on the comparison.
10 . The apparatus of claim 9 , wherein the processor circuitry is to:
when the speed is below the threshold speed, not analyze one or more frames of the video to identify the plurality of candidate signs; and
when the speed is above the threshold speed, create additional frames for analysis based on interpolation of the frames in the video.
11 . The apparatus of claim 1 , wherein to select one or more of the candidate signs as the performed signs, the processor circuitry is to:
identify at least a partial overlap in video segments corresponding to a first candidate sign and a second candidate sign;
identify a higher confidence score between a first confidence score of the first candidate sign and a second confidence score of the second candidate sign; and
select the first candidate sign or the second candidate sign with the higher confidence score as the performed sign.
12 . A non-transitory machine readable medium comprising instructions to cause one or more processors to perform operations to translate a real-time conversation between sign language and spoken language, the operations including:
obtain a continuous sign video that captures signs generated during the real-time conversation;
create a list of a plurality of random video segments of the continuous sign video, the plurality of video segments including random start and end times during a duration of the continuous sign video;
identify, via execution of a trained single sign language recognition model that is provided the continuous sign video and the list of random video segments, a plurality of candidate signs, each of the plurality of candidate signs identified based on features identified in respective subsets of the plurality of video segments, wherein at least two of the subsets of the plurality of video segments include one or more of the same video segments;
associate, via the execution of the trained single sign language recognition model, a respective gloss to each of the plurality of candidate signs;
group the glosses of the plurality of candidate signs that are identified using one or more of the same video segments to form a plurality of gloss groups;
rank the glosses within each of the plurality of gloss groups;
select one of the plurality of candidate signs as performed signs in each of the plurality of gloss groups based on the rank of the glosses in each of the plurality of gloss groups; and
generate, via a model, at least one of visual, text, or audio data based on the performed signs.
13 . The medium of claim 12 , wherein the operations further include transmit the audio data to a user device including a speaker for conversion of the audio data to a sound wave.
14 . The medium of claim 12 , wherein the operations further include:
identify non-manual elements in the continuous sign video; and
adjust the audio data, by adjusting at least one of a volume, cadence, tone, or emotion of speech in the audio data based on the non-manual elements.
15 . The medium of claim 12 , wherein the operations further include:
identify frames of the continuous sign video without sign language as non-useful frames; and
ignore the non-useful frames of the continuous sign video prior to identifying the plurality of candidate signs.
16 . The medium of claim 12 , wherein the operations further include:
detect a speed of signing in the video;
perform a comparison of the speed to a threshold speed; and
change a number of video frames in the plurality of video segments analyzed to identify the plurality of candidate signs based on the comparison.
17 . The medium of claim 16 , wherein the operations further include when the speed is below the threshold speed, not analyze one or more frames of the video to identify the plurality of candidate signs.
18 . The medium of claim 16 , wherein the operations further include when the speed is above the threshold speed, create additional frames for analysis based on interpolation of the frames in the video.
19 . A method to convert sign language to spoken language, the method comprising:
obtaining a continuous sign video that captures signs generated during a real-time conversation for which translation between sign language and spoken language is occurring;
creating a list of a plurality of random video segments of the continuous sign video, the plurality of video segments including random start and end times during a duration of the continuous sign video;
identifying, via execution of a trained single sign language recognition model that is provided the continuous sign video and the list of random video segments, a plurality of candidate signs, each of the plurality of candidate signs identified based on features identified in respective subsets of the plurality of video segments, wherein at least two of the subsets of the plurality of video segments include one or more of the same video segments;
associating, via the execution of the trained single sign language recognition model, a respective gloss each of the plurality of candidate signs;
generating, via the execution of the trained single sign language recognition model, a respective confidence score for each of the respective glosses, the confidence score indicating a probability identified by the trained single sign language recognition model that the respective glosses are an accurate translation of the plurality of candidate signs;
determining which of the plurality of candidate signs are identified using one or more of the same video segments; and
selecting performed signs from the plurality of candidate signs based on the respective confidence scores and which of the plurality of candidate signs are identified using one or more of the same video segments, the performed signs used for the translation between sign language and spoken language during the real-time conversation.
20 . The method of claim 19 , further including:
converting the performed signs to audio data; and
transmitting the audio data to a remote device for presentation as sound.