Systems and methods for sign language recognition
A sensory eyewear system for a mixed reality device can facilitate user's interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpret a sign language, and present the translated information to a user of the mixed reality device. The wearable system can also recognize text in the user's environment, modify the text (e.g., by changing the content or display characteristics of the text), and render the modified text to occlude the original text.
1. A computer-implemented method for sign language recognition, comprising:
under control of a wearable system comprising a head-mounted display configured to present virtual content to a user of the wearable system:
receiving image information captured by an imaging system of the wearable system in an environment;
detecting, by analyzing the image information and as detected sign language, a presence of sign language in the image information;
identifying a first source of the detected sign language;
determining a change in source to a second source of sign language has occurred;
translating sign language of the second source of sign language into a language understood by the user;
monitoring for a request for auxiliary information associated with the language understood by the user;
retrieving the auxiliary information associated with the language understood by the user; and
displaying, on the wearable system, the auxiliary information.
2. The computer-implemented method of claim 1 , comprising:
capturing audio information in the environment.
3. The computer-implemented method of claim 2 , wherein a presence of sign language in the image information includes one or more of a sequence of hand gestures and a sequence of hand gestures and lip movements unaccompanied by audio information.
4. The computer-implemented method of claim 2 , wherein detecting, by analyzing the image information and as detected sign language, a presence of sign language in the image information, comprises:
using contextual information from the environment to interpret the detected sign language.
5. The computer-implemented method of claim 1 , wherein identifying a first source of the detected sign language, comprises:
processing sign language from a person in or near a center of a user's field-of-view or a person at whom the user is looking; and
if other persons are detected to be currently conversing in sign language, discarding processing of sign language associated with the other persons.
6. The computer-implemented method of claim 1 , wherein translating sign language of the second source of sign language into a language understood by the user, comprises:
converting the sign language of the second source of sign language into text; or
displaying the sign language of the second source of sign language into a graphic representation of a different sign language.
7. The computer-implemented method of claim 1 , comprising:
receiving, from the wearable system, the request for auxiliary information in response to a triggering condition, wherein the triggering condition includes a user's gesture, a user's actuation of a user input device, or a detection of an unknown word or phrase.
8. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations for sign language recognition, comprising:
under control of a wearable system comprising a head-mounted display configured to present virtual content to a user of the wearable system:
receiving image information captured by an imaging system of the wearable system in an environment;
detecting, by analyzing the image information and as detected sign language, a presence of sign language in the image information;
identifying a first source of the detected sign language;
determining a change in source to a second source of sign language has occurred;
translating sign language of the second source of sign language into a language understood by the user;
monitoring for a request for auxiliary information associated with the language understood by the user;
retrieving the auxiliary information associated with the language understood by the user; and
displaying, on the wearable system, the auxiliary information.
9. The non-transitory, computer-readable medium of claim 8 , comprising:
capturing audio information in the environment.
10. The non-transitory, computer-readable medium of claim 9 , wherein a presence of sign language in the image information includes one or more of a sequence of hand gestures and a sequence of hand gestures and lip movements unaccompanied by audio information.
11. The non-transitory, computer-readable medium of claim 9 , wherein detecting, by analyzing the image information and as detected sign language, a presence of sign language in the image information, comprises:
using contextual information from the environment to interpret the detected sign language.
12. The non-transitory, computer-readable medium of claim 8 , wherein identifying a first source of the detected sign language, comprises:
processing sign language from a person in or near a center of a user's field-of-view or a person at whom the user is looking; and
if other persons are detected to be currently conversing in sign language, discarding processing of sign language associated with the other persons.
13. The non-transitory, computer-readable medium of claim 8 , wherein translating sign language of the second source of sign language into a language understood by the user, comprises:
converting the sign language of the second source of sign language into text; or
displaying the sign language of the second source of sign language into a graphic representation of a different sign language.
14. The non-transitory, computer-readable medium of claim 8 , comprising:
receiving, from the wearable system, the request for auxiliary information in response to a triggering condition, wherein the triggering condition includes a user's gesture, a user's actuation of a user input device, or a detection of an unknown word or phrase.
15. A computer-implemented system for sign language recognition, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations, comprising:
under control of a wearable system comprising a head-mounted display configured to present virtual content to a user of the wearable system:
receiving image information captured by an imaging system of the wearable system in an environment;
detecting, by analyzing the image information and as detected sign language, a presence of sign language in the image information;
identifying a first source of the detected sign language;
determining a change in source to a second source of sign language has occurred;
translating sign language of the second source of sign language into a language understood by the user;
monitoring for a request for auxiliary information associated with the language understood by the user;
retrieving the auxiliary information associated with the language understood by the user; and
displaying, on the wearable system, the auxiliary information.
16. The computer-implemented system of claim 15 , comprising:
capturing audio information in the environment.
17. The computer-implemented system of claim 16 , wherein a presence of sign language in the image information includes one or more of a sequence of hand gestures and a sequence of hand gestures and lip movements unaccompanied by audio information.
18. The computer-implemented system of claim 16 , wherein detecting, by analyzing the image information and as detected sign language, a presence of sign language in the image information, comprises:
using contextual information from the environment to interpret the detected sign language.
19. The computer-implemented system of claim 15 , wherein identifying a first source of the detected sign language, comprises:
processing sign language from a person in or near a center of a user's field-of-view or a person at whom the user is looking; and
if other persons are detected to be currently conversing in sign language, discarding processing of sign language associated with the other persons.
20. The computer-implemented system of claim 15 , wherein translating sign language of the second source of sign language into a language understood by the user, comprises:
converting the sign language of the second source of sign language into text; or
displaying the sign language of the second source of sign language into a graphic representation of a different sign language.
21. The computer-implemented system of claim 15 , comprising:
receiving, from the wearable system, the request for auxiliary information in response to a triggering condition, wherein the triggering condition includes a user's gesture, a user's actuation of a user input device, or a detection of an unknown word or phrase.