Sign language communication with communication devices
View Patent ↗Implementations enable conversations between operators of communication devices who use sign language and other operators who don't. A method may include receiving images of first sign language gestures captured by a camera of a first communication device, converting the first sign language gestures into first text, transmitting the first text to a second communication device, receiving second text from the second communication device, and converting the second text into images of second sign language gestures made by an avatar. The method may also include operating the camera to capture the images of the first sign language gestures and presenting the images of the second sign language gestures on a display of the first communication device. The method may further include receiving first speech captured at the second communication device, converting the first speech into third text, and then into images of third sign language gestures made by the avatar.
1. A computer-implemented method to enable sign language communication through communication devices, the method comprising:
receiving, by a processor, images of first sign language gestures captured by a camera of a first communication device;
converting, by the processor, the first sign language gestures into first text using training data stored within a storage of the first communication device, wherein the training data is generated at least from a corrected interpretation of a training sequence of sign language gestures;
transmitting the first text to a second communication device;
receiving second text from the second communication device; and
converting, by the processor, the second text into images of second sign language gestures made by an avatar, wherein the images of the second sign language gestures comprise motion video animation generated from a library of sequences of poses that, when presented in sequence at a predetermined frame rate, provide animated movement of the avatar forming sign language gestures.
2. The computer-implemented method of claim 1 , the method further comprises:
operating the camera to capture the images of the first sign language gestures; and
presenting the images of the second sign language gestures on a display of the first communication device.
3. The computer-implemented method of claim 2 , further comprising generating the training data that correlates third sign language gestures to the corrected interpretation of the third sign language gestures, wherein the generating the training data comprises:
operating the camera to capture images of the third sign language gestures;
interpreting the third sign language gestures;
receiving, at the first communication device, the corrected interpretation of the third sign language gestures; and
storing the training data within the storage of the first communication device for use in the conversion of the first sign language gestures into the first text.
4. The computer-implemented method of claim 1 , further comprising:
receiving, by the processor, first speech captured at the second communication device;
converting, by the processor, the first speech into third text; and
converting, by the processor, the third text into images of third sign language gestures made by the avatar.
5. The computer-implemented method of claim 4 , further comprising:
converting, by the processor, the first sign language gestures into second speech; and
transmitting the second speech to the second communication device.
6. The computer-implemented method of claim 1 , the method further comprises:
receiving the images of the first sign language gestures from the first communication device via a network; and
transmitting the images of the second sign language gestures to the first communication device via the network to enable presentation of the images of the second sign language gestures on a display of the first communication device.
7. The computer-implemented method of claim 6 , further comprising:
receiving, at a connection server, a request from one of the first communication device and the second communication device to communicate with another of the first communication device and the second communication device;
receiving, at the connection server, indications from the first communication device that the first communication device is operated by an operator who uses sign language and that the first communication device comprises the camera and the display; and
in response to the request and the indications, routing communications between the first and second communication devices, performing the conversion from the first sign language gestures into the first text at the connection server, and performing the conversion from the second text to the second sign language gestures at the connection server.
8. The computer-implemented method of claim 6 , further comprising:
receiving, at a connection server, indications from each communication device of a plurality of communication devices that each communication device of the plurality of communication devices is associated with an organization and that each communication device of the plurality of communication devices is operated by an operator who does not use sign language;
receiving, at the connection server, a request from the first communication device to communicate with a communication device of the plurality of communication devices;
receiving, at the connection server, indications from the first communication device that the first communication device is operated by an operator who uses sign language and that the first communication device comprises the camera and the display;
determining that none of the communication devices of the plurality of communication devices is operated by an operator who uses sign language; and
in response to the request, the indications and the determination, selecting a communication device from among the plurality of communication devices that is indicated as available and operated by an available operator as the second communication device and routing communications between the first communication device and the selected communication device.
9. A non-transitory machine-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to:
receive images of first sign language gestures captured by a camera of a first communication device;
convert the first sign language gestures into first text using training data, wherein the training data is based on a training sequence of sign language gestures and includes a corrected interpretation of the training sequence of sign language gestures;
transmit the first text to a second communication device;
receive second text from the second communication device; and
convert the second text into images of second sign language gestures made by an avatar, wherein the images of the second sign language gestures comprise motion video animation generated from a library of sequences of poses that, when presented in sequence at a predetermined frame rate, provide animated movement of the avatar forming sign language gestures.
10. The non-transitory machine-readable medium of claim 9 , wherein the processor is further caused to:
operate the camera to capture the images of the first sign language gestures; and
present the images of the second sign language gestures on a display of the first communication device.
11. The non-transitory machine-readable medium of claim 10 , wherein the processor is further caused to generate the training data that correlates third sign language gestures to the corrected interpretation of the third sign language gestures, wherein the processor is further caused to:
operate the camera to capture images of the third sign language gestures;
interpret the third sign language gestures;
receive the corrected interpretation of the third sign language gestures; and
store the training data within the storage of the first communication device for use in the conversion of the first sign language gestures into the first text.
12. The non-transitory machine-readable medium of claim 9 , wherein the processor is further caused to:
receive first speech captured at the second communication device;
convert the first speech into third text; and
convert the third text into images of third sign language gestures made by the avatar.
13. The non-transitory machine-readable medium of claim 12 , wherein the processor is further caused to:
convert the first sign language gestures into second speech; and
transmit the second speech to the second communication device.
14. The non-transitory machine-readable medium of claim 9 , wherein the processor is further caused to:
receive the images of the first sign language gestures from the first communication device via a network; and
transmit the images of the second sign language gestures to the first communication device via the network to enable presentation of the images of the second sign language gestures on a display of the first communication device.
15. The non-transitory machine-readable medium of claim 14 , wherein the processor is further caused to:
receive a request from one of the first communication device and the second communication device to communicate with another of the first communication device and the second communication device;
receive indications from the first communication device that the first communication device is operated by an operator who uses sign language and that the first communication device comprises the camera and the display; and
in response to the request and the indications, route communications between the first and second communication devices, perform the conversion from the first sign language gestures into the first text, and perform the conversion from the second text to the second sign language gestures.
16. An apparatus to enable sign language communication through communication devices, the apparatus comprising:
a processor; and
a storage coupled to the processor to store instructions, that when executed by the processor cause the processor to:
receive images of first sign language gestures captured by a camera of a first communication device;
convert the first sign language gestures into first text using training data, wherein the training data is based on a training sequence of sign language gestures and includes a corrected interpretation of the training sequence of sign language gestures;
transmit the first text to a second communication device;
receive second text from the second communication device; and
convert the second text into images of second sign language gestures made by an avatar, wherein the images of the second sign language gestures comprise motion video animation generated from a library of sequences of poses that, when presented in sequence at a predetermined frame rate, provide animated movement of the avatar forming sign language gestures.
17. The apparatus of claim 16 , wherein the processor is further caused to:
operate the camera to capture the images of the first sign language gestures; and
present the images of the second sign language gestures on a display of the first communication device.
18. The apparatus of claim 16 , wherein the processor is further caused to:
receive first speech captured at the second communication device;
convert the first speech into third text; and
convert the third text into images of third sign language gestures made by the avatar.
19. The apparatus of claim 18 , wherein the processor is further caused to:
convert the first sign language gestures into second speech; and
transmit the second speech to the second communication device.
20. The apparatus of claim 16 , wherein the processor is further caused to:
receive the images of the first sign language gestures from the first communication device via a network; and
transmit the images of the second sign language gestures to the first communication device via the network to enable presenting of the images of the second sign language gestures on a display of the first communication device.