Natural interactive user interface using artificial intelligence and freeform input
In an example embodiment, first user input including handwriting input and non-alphanumeric symbolic input is detected. The non-alphanumeric symbolic input is input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions. A combination of the action having the highest probability score and textual input from the handwriting input is input into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services. The combination of the textual input and the selected action is transformed into a native request for the selected service based on the service model for the selected service.
1. A device comprising:
an operating system executed by a hardware processor,
a graphical user interface component, when executing on the operating system, configured to detect a first user input including both handwriting input and non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input;
a handwriting recognition component, when executing on the operating system, configured to perform handwriting recognition on the handwriting input to transform the handwriting input into a textual input;
a first machine learning model, when executing on the operating system, trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action;
a second machine learning model, when executing on the operating system, trained to select a service from a plurality of services based on the textual input and a selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network; and
the graphical user interface component configured to:
display a selection of an action in the set of possible actions output by the first machine learning model having a highest probability score;
input a combination of the textual input and the selected action into the second machine learning model;
transform the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and
send the native request to the selected service for handling by the selected service.
2. The system of claim 1 , wherein the non-alphanumeric symbolic input includes gesture-based input.
3. The system of claim 1 , wherein the non-alphanumeric symbolic input includes a drawing made by the user via the graphical user interface.
4. The system of claim 1 , wherein the non-alphanumeric symbolic input includes circling an object rendered in the graphical user interface and connecting the circled object to the handwriting input.
5. The system of claim 1 , wherein the second machine learning model is coupled to a plurality of different semantic models of services, each semantic model of a service defining how the corresponding service is used, its domain, and its construction, and containing one or more native requests for the corresponding service.
6. The system of claim 5 , wherein each of the one or more native requests has a set of input parameters, and wherein the second machine learning model is further trained to output a selected native request for the selected service based on the set of input parameters for the selected native request.
7. The system of claim 1 , wherein each of the plurality of services is located on a different data source.
8. A method comprising:
detecting first user input, the first user input including both a handwriting input and a non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input;
transforming the handwriting input into textual input;
inputting the non-alphanumeric symbolic input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and to output a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action;
selecting an action in the set of possible actions output by the first machine learning model having a highest probability score;
inputting a combination of the textual input and the selected action into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network;
transforming the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and
sending the native request to the selected service for handling by the selected service.
9. The method of claim 8 , wherein the non-alphanumeric symbolic input includes gesture-based input.
10. The method of claim 8 , wherein the non-alphanumeric symbolic input includes a drawing made by the user via a graphical user interface.
11. The method of claim 8 , wherein the non-alphanumeric symbolic input includes circling an object rendered in a graphical user interface and connecting the circled object to the handwriting input.
12. The method of claim 8 , wherein the second machine learning model is coupled to a plurality of different semantic models of services, each semantic model of a service defining how the corresponding service is used, its domain, and its construction, and containing one or more native requests for the corresponding service.
13. The method of claim 12 , wherein each of the one or more native requests has a set of input parameters, and wherein the second machine learning model is further trained to output a selected native request for the selected service based on the set of input parameters for the selected native request.
14. The method of claim 8 , wherein each of the plurality of services is located on a different data source.
15. A non-transitory machine-readable storage medium comprising instructions which, when implemented by one or more machines, cause the one or more machines to perform operations comprising:
detecting first user input, the first user input including both handwriting input and non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input;
transforming the handwriting input into textual input;
inputting the non-alphanumeric symbolic input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action;
selecting an action in the set of possible actions output by the first machine learning model having a highest probability score;
inputting a combination of the textual input and the selected action into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network;
transforming the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and
sending the native request to the selected service for handling by the selected service.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the non-alphanumeric symbolic input includes gesture-based input.
17. The non-transitory machine-readable storage medium of claim 15 , wherein the non-alphanumeric symbolic input includes a drawing made by the user via a graphical user interface.
18. The non-transitory machine-readable storage medium of claim 15 , wherein the non-alphanumeric symbolic input includes circling an object rendered in a graphical user interface and connecting the circled object to the handwriting input.
19. The non-transitory machine-readable storage medium of claim 15 , wherein the second machine learning model is coupled to a plurality of different semantic models of services, each semantic model of a service defining how the corresponding service is used, its domain, and its construction, and containing one or more native requests for the corresponding service.
20. The non-transitory machine-readable storage medium of claim 19 , wherein each of the one or more native requests has a set of input parameters, and wherein the second machine learning model is further trained to output a selected native request for the selected service based on the set of input parameters for the selected native request.