VPA with integrated object recognition and facial expression recognition
Methods, computing devices, and computer-program products are provided for implementing a virtual personal assistant. In various implementations, a virtual personal assistant can be configured to receive sensory input, including at least two different types of information. The virtual personal assistant can further be configured to determine semantic information from the sensory input, and to identify a context-specific framework. The virtual personal assistant can further be configured to determine a current intent. Determining the current intent can include using the semantic information and the context-specific framework. The virtual personal assistant can further be configured to determine a current input state. Determining the current input state can include using the semantic information and one or more behavioral models. The behavioral models can include one or more interpretations of previously-provided semantic information. The virtual personal assistant can further be configured to determine an action using the current intent and the current input state.
1. A method, comprising:
receiving, by a computing device, sensory input, wherein the sensory input includes at least two different types of information;
determining semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;
determining an action, wherein determining the action includes using the semantic information;
identifying a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;
determining a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;
determining a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information;
using one or more of the current intent and the current input state to modify the action prior to outputting the action to a user of the computing device, wherein to modify the action comprises changing a question type from a first type designed to elicit a first type of answer to a second type designed to elicit a second type of answer.
2. The method of claim 1 , wherein types of information include speech information, graphical information, audio input, image input, or tactile input.
3. The method of claim 1 , wherein an emotional state is capable of being derived from audio input.
4. The method of claim 1 , wherein physical gestures are capable of being derived from image input.
5. The method of claim 1 , wherein an emotional state is capable of being derived from image input.
6. The method of claim 1 , wherein behavioral models include a configurable preference model, and wherein the configurable preference model includes one or more previous input states and semantic information associated with the one or more previous input states.
7. The method of claim 1 , wherein the context-specific framework includes a dynamic ontology, wherein the dynamic ontology includes a cumulative sequence of one or more previous intents derived from a context-specific ontology.
8. The method of claim 1 , wherein to modify the action comprises increasing a volume of system output or decreasing the volume of system output or increasing a rate of system output or decreasing the rate of system output.
9. A virtual personal assistant device, comprising:
one or more processors; and
a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving, by a computing device, sensory input, wherein the sensory input includes at least two different types of information;
determining semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;
determining an action, wherein determining the action includes using the semantic information;
identifying a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;
determining a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;
determining a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information; and
using one or more of the current intent and the current input state to modify the action prior to outputting the action to a user of the computing device, wherein to modify the action comprises changing a question type from a first type designed to elicit a first type of answer to a second type designed to elicit a second type of answer.
10. The virtual personal assistant device of claim 9 , wherein types of information include speech information, graphical information, audio input, image input, or tactile input.
11. The virtual personal assistant device of claim 9 , wherein an emotional state is capable of being derived from audio input.
12. The virtual personal assistant device of claim 9 , wherein physical gestures are capable of being derived from image input.
13. The virtual personal assistant device of claim 9 , wherein an emotional state is capable of being derived from image input.
14. The virtual personal assistant device of claim 9 , wherein behavioral models include a configurable preference model, and wherein the configurable preference model includes one or more previous input states and semantic information associated with the one or more previous input states.
15. The virtual personal assistant device of claim 9 , wherein the context-specific framework includes a dynamic ontology, wherein the dynamic ontology includes a cumulative sequence of one or more previous intents derived from a context-specific ontology.
16. The virtual personal assistant device of claim 9 , wherein to modify the action comprises causing the action to change from an open-ended dialog approach to a yes/no questioning approach or from a yes/no questioning approach to an open-ended dialog approach, or increasing a volume of system output or decreasing the volume of system output, or increasing a rate of system output or decreasing the rate of system output.
17. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions that, when executed by one or more processors, cause the one or more processors to:
receive, by a computing device, sensory input, wherein the sensory input includes at least two different types of information;
determine semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;
determine an action, wherein determining the action includes using the semantic information;
determine an action, wherein determining the action includes using the semantic information;
identify a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;
determine a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;
determine a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information; and
use one or more of the current intent and the current input state to modify the action prior to outputting the action to a user of the computing device, wherein to modify the action comprises changing the action from an open-ended dialog approach to a yes/no questioning approach or from a yes/no questioning approach to an open-ended dialog approach or from a first question not specifically designed to elicit easy to remember information to a second question designed to elicit easy to remember information.
18. The computer-program product of claim 17 , wherein types of information include speech information, graphical information, audio input, image input, or tactile input.
19. The computer-program product of claim 17 , wherein an emotional state is capable of being derived from audio input.
20. The computer-program product of claim 17 , wherein physical gestures are capable of being derived from image input.
21. The computer-program product of claim 17 , wherein an emotional state is capable of being derived from image input.
22. The computer-program product of claim 17 , wherein behavioral models include a configurable preference model, and wherein the configurable preference model includes one or more previous input states and semantic information associated with the one or more previous input states.
23. The computer-program product of claim 17 , wherein to modify the action comprises causing the action to change from an open-ended dialog approach to a yes/no questioning approach or from a yes/no questioning approach to an open-ended dialog approach, or increasing a volume of system output or decreasing the volume of system output, or increasing a rate of system output or decreasing the rate of system output.