VPA with integrated object recognition and facial expression recognition
Methods, computing devices, and computer-program products are provided for implementing a virtual personal assistant. In various implementations, a virtual personal assistant can be configured to receive sensory input, including at least two different types of information. The virtual personal assistant can further be configured to determine semantic information from the sensory input, and to identify a context-specific framework. The virtual personal assistant can further be configured to determine a current intent. Determining the current intent can include using the semantic information and the context-specific framework. The virtual personal assistant can further be configured to determine a current input state. Determining the current input state can include using the semantic information and one or more behavioral models. The behavioral models can include one or more interpretations of previously-provided semantic information. The virtual personal assistant can further be configured to determine an action using the current intent and the current input state.
1. A method comprising:
receiving, by a computing device, sensory input comprising at least audio input and visual input;
determining semantic information from the sensory input;
determining a first questioning style to question a user using the semantic information;
determining scene information from the visual input;
determining, after waiting for audio input from the user, an input state associated with the user using the audio input and the scene information;
changing the first questioning style to a second questioning style using the determined input state, the second questioning style being different from the first questioning style; and
outputting a question to the user using the second questioning style.
2. The method of claim 1 , further comprising extracting the scene information from at least one image input or at least one video input.
3. The method of claim 1 , further comprising determining object information from the scene information and using the object information to determine the input state.
4. The method of claim 1 , further comprising determining event information from the scene information and using the event information to determine the input state.
5. The method of claim 1 , further comprising determining motion information from the scene information and using the motion information to determine the input state.
6. The method of claim 1 , further comprising determining the scene information using at least one machine learned model trained using domain-specific training samples including at least one image sample or at least one video sample.
7. The method of claim 1 , further comprising, using the scene information, determining a temporal sequence and using the temporal sequence to determine the input state, wherein the temporal sequence comprises (i) an object, (ii) an event, or (iii) a combination of (i) and (ii).
8. The method of claim 1 , further comprising extracting an image of an iris of a human eye from the visual input, using the image to generate a code, comparing the code to a template, determining biometric information based on a comparison of the code to the template, and using the biometric information to determine the input state.
9. The method of claim 1 , wherein the input state is at least one of an emotional state, cognitive state, or mental state of the user.
10. The method of claim 1 , wherein the sensory input is generated by the user.
11. A device, comprising: at least one processor;
at least one sensor coupled to the at least one processor; and memory coupled to the at least one processor;
the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations comprising:
receiving, by the at least one sensor, audio input;
receiving, by the at least one sensor, visual input;
determining semantic information from the audio input;
determining a first questioning style to question a user using the semantic information;
determining scene information from the visual input;
determining, after waiting for audio input from the user, an input state associated with the user using the audio input and the scene information;
changing the first questioning style to a second questioning style using the determined input state, the second questioning style being different from the first questioning style; and
outputting a question to the user using the second questioning style.
12. The device of claim 11 , the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations further comprising extracting the scene information from at least one image input or at least one video input.
13. The device of claim 11 , the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations further comprising determining object information from the scene information and using the object information to determine the input state.
14. The device of claim 11 , the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations further comprising determining event information from the scene information and using the event information to determine the input state.
15. The device of claim 11 , the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations further comprising determining motion information from the scene information and using the motion information to determine the input state.
16. The device of claim 11 , the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations further comprising determining the input state using at least one machine-learned model trained using domain-specific training samples including at least one image sample or at least one video sample.
17. The device of claim 11 , the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations further comprising, using the scene information, determining a temporal sequence and using the temporal sequence to determine the input state, wherein the temporal sequence comprises (i) at least one object, (ii) at least one event, or (iii) a combination of (i) and (ii).
18. The device of claim 11 , the memory cooperating with the at least one processor to cause the at least one processor to be capable of performing operations further comprising extracting an image of an iris of a human eye from the visual input, using the image to generate a code, comparing the code to a template, determining biometric information based on a comparison of the code to the template, and using the biometric information to determine the input state.
19. At least one memory device storing instructions which, when executed by at least one processor, cause the at least one processor to be capable of performing operations comprising:
receiving an audio input;
receiving a visual input;
determining semantic information from the audio input;
determining a first questioning style to question a user using the semantic information;
determining scene information from the visual input;
determining, after waiting for audio input from the user, an input state associated with the user using the audio input and the scene information;
changing the first questioning style to a second questioning style using the determined input state, the second questioning style being different from the first questioning style; and
outputting a question to the user using the second questioning style.
20. The at least one memory device of claim 19 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to be capable of performing operations further comprising extracting the scene information from at least one image input or at least one video input and determining the input state using at least one machine-learned model trained using domain-specific training samples including at least one image sample or at least one video sample.
21. The at least one memory device of claim 19 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to be capable of performing operations further comprising determining object information from the scene information and using the object information to determine the input state.
22. The at least one memory device of claim 19 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to be capable of performing operations further comprising, using the scene information, determining a temporal sequence and using the temporal sequence to determine the input state, wherein the temporal sequence comprises (i) at least one object, (ii) at least one event, or (iii) a combination of (i) and (ii).