Systems and methods for recognition of sign language for improved viewing experiences
Systems and methods are described herein for selecting a closed captioning language track. Control circuitry may capture an image or video of a user while the user is performing a gesture of a sign language. The control circuitry may access a database comprising a plurality of entries corresponding to a plurality of gestures in respective sign languages. The control circuitry may compare a first pixel map corresponding to the image or video of the user with pixel maps corresponding to each of the plurality of entries in order to identify a preferred sign language. The control circuitry may receive metadata associated with a media asset comprising a plurality of closed captioning language tracks, each closed captioning language track comprising closed captioning for the media asset in a different language and select one of the closed captioning language tracks that corresponds to the preferred sign language.
1. A method for selecting a closed captioning language track, the method comprising:
capturing an image or video of a user while the user is performing a first gesture of a sign language, the image or video of the user comprising a first pixel map;
accessing a database comprising a plurality of entries, each entry comprising a pixel map that corresponds to a gesture in a respective sign language and an indication of the respective sign language;
comparing the first pixel map to the pixel map of each of the plurality of entries, comprising, for each respective entry:
extracting a second pixel map from the respective entry;
calculating an average pixel value for the second pixel map;
normalizing the first pixel map using the calculated average pixel value;
comparing pixels of the first pixel map to pixels of the second pixel map;
identifying a pattern of pixels from the first pixel map that is within a threshold value of corresponding pixels from the second pixel map;
identifying, based on the comparison, an entry of the plurality of entries corresponding to a second gesture that matches the first gesture;
receiving metadata associated with a media asset, the metadata comprising a plurality of closed captioning language tracks, each closed captioning language track comprising closed captioning for the media asset in a different language; and
selecting one of the closed captioning language tracks that corresponds to a sign language indicated in the identified entry.
2. The method of claim 1 , further comprising generating for display the media asset with the selected closed captioning language track.
3. The method of claim 1 , further comprising generating for display an alert which includes an indication of the selected closed captioning language track.
4. The method of claim 1 , wherein capturing the image or video of the user occurs while the media asset is being generated for display.
5. The method of claim 1 , further comprising:
identifying, based on the comparison, a second entry of the plurality of entries corresponding to a third gesture that matches the first gesture; and
generating for display an alert including user-selectable options indicating two of the closed captioning language tracks that correspond to sign languages indicated in the identified entries; and
wherein selecting one of the closed captioning language tracks comprises receiving a user selection of one of the options in the alert.
6. The method of claim 1 , further comprising storing the selected closed captioning language track to a user profile.
7. The method of claim 1 , wherein the gesture comprises one or a combination of the following: a hand shape, a hand orientation, a movement of the hand, arm, or body, a lip movement, or a facial expression.
8. The method of claim 1 , wherein the plurality of different sign language libraries comprise libraries for one or more of the following: American sign language, Pidgin Signed English, Signing Exact English, Spanish sign language, Catalan sign language, French sign language, German sign language, British sign language, Chinese sign language, Japanese sign language, Irish sign language, Auslan, Swedish sign language, and finger spelling.
9. The method of claim 1 , wherein comparing the identified pixels of the first pixel map with the second pixel map comprises scaling one of the first pixel map or the second pixel map.
10. The method of claim 1 , wherein each of the plurality of entries comprises an indication of one or more associated spoken and/or written languages.
11. A system for selecting a closed captioning language track, the system comprising:
a storage device;
a database comprising a plurality of entries, each entry comprising a pixel map that corresponds to a gesture in a respective sign language and an indication of the respective sign language;
image detection circuitry configured to:
capture an image or video of a user while the user is performing a first gesture of a sign language, the image or video of the user comprising a first pixel map; and
store the first pixel map on the storage device;
control circuitry configured to:
access the database to retrieve the pixel map of each of the plurality of entries;
retrieve the first pixel map from the storage device;
compare the first pixel map to the pixel map of each of the plurality of entries, comprising, for each respective entry:
extracting a second pixel map from the respective entry;
calculating an average pixel value for the second pixel map;
normalizing the first pixel map using the calculated average pixel value;
comparing pixels of the first pixel map to pixels of the second pixel map;
identifying a pattern of pixels from the first pixel map that is within a threshold value of corresponding pixels from the second pixel map;
identify, based on the comparison, an entry of the plurality of entries corresponding to a second gesture that matches the first gesture;
receive metadata associated with a media asset, the metadata comprising a plurality of closed captioning language tracks, each closed captioning language track comprising closed captioning for the media asset in a different language; and
select one of the closed captioning language tracks that corresponds to a sign language indicated in the identified entry.
12. The system of claim 11 , wherein the control circuitry is further configured to generate for display the media asset with the selected closed captioning language track.
13. The system of claim 11 , wherein the control circuitry is further configured to generate for display an alert which includes an indication of the selected closed captioning language track.
14. The system of claim 11 , wherein the control circuitry is further configured to capture the image or video of the user occurs while the media asset is being generated for display.
15. The system of claim 11 , wherein the control circuitry is further configured to:
identify, based on the comparison, a second entry of the plurality of entries corresponding to a third gesture that matches the first gesture; and
generate for display an alert including user-selectable options indicating two of the closed captioning language tracks that correspond to sign languages indicated in the identified entries; and
wherein selecting one of the closed captioning language tracks comprises receiving a user selection of one of the options in the alert.
16. The system of claim 11 , wherein the control circuitry is further configured to store the selected closed captioning language track to a user profile.
17. The system of claim 11 , wherein the gesture comprises one or a combination of the following: a hand shape, a hand orientation, a movement of the hand, arm, or body, a lip movement, or a facial expression.
18. The system of claim 11 , wherein the plurality of different sign language libraries comprise libraries for one or more of the following: American sign language, Pidgin Signed English, Signing Exact English, Spanish sign language, Catalan sign language, French sign language, German sign language, British sign language, Chinese sign language, Japanese sign language, Irish sign language, Auslan, Swedish sign language, and finger spelling.
19. The system of claim 11 , wherein the control circuitry is configured to compare the identified pixels of the first pixel map with the second pixel map by scaling one of the first pixel map or the second pixel map.
20. The system of claim 11 , wherein each of the plurality of entries comprises an indication of one or more associated spoken and/or written languages.