Image based command classification and task engine for a computing system
Provided are methods, systems, and computer storage media for determining a command (e.g., intent) of an image based on image data features. A task associated with the determined command is generated based on a portion of the image data features. Task entities corresponding to the task are determined. The task and the corresponding task entities are generated and configured for use in a computer productivity application. Accordingly, present embodiments provide for generating command-specific tasks and task entities that may be integratable for use in a computer productivity application.
1. A computerized system, the computerized system comprising:
at least one computer processor; and
computer memory storing computer-readable instructions that, when used by the at least one computer processor, cause the at least one computer processor to perform operations comprising:
receiving an image comprising at least an indication of non-alphanumeric-character objects;
extracting image data for the image based on at least the non-alphanumeric-character objects to produce a plurality of image data features;
determining, by a machine learning model, an image context based on a first image data feature of the plurality of image data features, wherein the image context defines a relationship between alphanumeric characters depicted in the image and the non-alphanumeric-character objects;
determining a command associated with the image based on a second image data feature in the plurality of image data features;
determining at least one task corresponding to the command based on at least one of: the command, the first image data feature, the image context, and the second image data feature;
based on the at least one task and the plurality of image data features, determining at least one task entity; and
generating the at least one task, comprising the at least one task entity and configured for use in a computer productivity application.
2. The computerized system of claim 1 , wherein the command is determined based on the machine learning model that is trained using a plurality of images comprising the alphanumeric characters and the non-alphanumeric-character objects.
3. The computerized system of claim 1 , wherein the image further comprises an indication of the alphanumeric characters, and wherein the plurality of image data features comprise visual features associated with text corresponding to the alphanumeric characters and spatial features associated with coordinates associated with and relating the alphanumeric characters and the non-alphanumeric-character objects.
4. The computerized system of claim 1 , wherein the image further comprises an indication of the alphanumeric characters, and the operations further comprising determining a corresponding coordinate set for each alphanumeric character and for each non-alphanumeric-character object, wherein the command is determined based on the second image data feature indicative of a comparison of each corresponding coordinate set to one another.
5. The computerized system of claim 1 , wherein the operations further comprise communicating the at least one task to an application layer of a computing device, wherein communicating the at least one task to the application layer comprises integrating the at least one task or the at least one task entity into a software application.
6. The computerized system of claim 5 , wherein the command comprises a calendar event, the at least one task entity comprises a calendar event description, the at least one task comprises creating the calendar event integratable with the software application, wherein communicating the at least one task to the application layer comprises adding the calendar event to an electronic calendar associated with the software application.
7. The computerized system of claim 1 , wherein the operations further comprise:
determining a second command of the image based on extracted text and extracted coordinates being applied to the machine learning model;
determining a second task based on at least one of the second command;
determining a second task entity associated with the second task based on the extracted text, the extracted coordinates, or both; and
generating the second task, comprising the second task entity.
8. The computerized system of claim 1 , wherein determining the command of the image comprises determining that the image comprises at least one of: a recipe, a scheduled event, a list of items.
9. The computerized system of claim 1 , wherein the image comprises a digital video or computer animation, and wherein the plurality of image data features comprise time-sensitive spatial image features and visual image features.
10. At least one computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to:
receive an image comprising alphanumeric characters and non-alphanumeric-character objects;
extract image data for the image based on the alphanumeric characters and the non-alphanumeric-character objects to produce a plurality of image data features;
determining a relationship between the alphanumeric characters and the non-alphanumeric-character objects based on a first image data feature of the plurality of image data features;
determine a command associated with the image based on a second image data feature in the plurality of image data features;
determine at least one task corresponding to the command based on at least one of: the command, the first image data feature, the second image data feature, and the relationship between the alphanumeric characters and the non-alphanumeric-character objects;
based on the at least one task and the plurality of image data features, determine at least one task entity; and
generate the at least one task, comprising the at least one task entity and configured for use in a computer productivity application.
11. The computer-storage media of claim 10 , wherein the plurality of image data features comprise visual features associated with text corresponding to the alphanumeric characters and spatial features associated with coordinates associated with and relating the alphanumeric characters and the non-alphanumeric-character objects.
12. The computer-storage media of claim 10 , wherein the instructions further cause the processor to determine a corresponding coordinate set for each alphanumeric character and for each non-alphanumeric-character object, wherein the command is determined based on the second image data feature indicative of a comparison of each corresponding coordinate set to one another.
13. The computer-storage media of claim 10 , wherein the instructions further cause the processor to communicate the at least one task to an application layer of a computing device, wherein communicating the at least one task to the application layer comprises integrating the at least one task or the at least one task entity into a software application.
14. The computer-storage media of claim 13 , wherein the command comprises a calendar event, the at least one task entity comprises an event description, the at least one task comprises the calendar event integratable with the software application, wherein communicating the at least one task to the application layer comprises appending the calendar event to the software application.
15. The computer-storage media of claim 10 , wherein the instructions further cause the processor to:
determine a second command of the image based on extracted text and extracted coordinates being applied to a machine learning model;
determine a second entity associated with the second command based on the extracted text, the extracted coordinates, or both; and
generate a second task, comprising the second entity, based on the second command.
16. A computer-implemented method, comprising:
receiving an image comprising alphanumeric characters and non-alphanumeric-character objects;
extracting, using a machine learning model, (i) text from the alphanumeric characters and (ii) spatial data from the alphanumeric characters and the non-alphanumeric-character objects, wherein (i) the text and (ii) the spatial data are associated with the machine learning model that is trained based on visual features associated with the text and based on spatial features associated with the spatial data;
determining, by the machine learning model, an image context based on the visual features and the spatial features, wherein the image context defines a relationship between the alphanumeric characters and the non-alphanumeric-character objects depicted in the image;
determining a command of the image based on the extracted text and the extracted spatial data being applied to the machine learning model;
determining a task associated with the command based on at least one of: the extracted text, the extracted spatial data, the command, and the image context;
determining at least one task entity corresponding to the task based on at least one of the extracted text, the extracted spatial data, the task, or the command; and
generating the task, comprising the at least one task entity.
17. The computer-implemented method of claim 16 , further comprising:
determining a second command of the image based on the extracted text and the extracted spatial data being applied to the machine learning model;
determining a second task entity associated with the second command based on the extracted text, the extracted spatial data, or both; and
generating a second task comprising the second task entity.
18. The computer-implemented method of claim 16 , further comprising communicating the task to an application layer of a computing device, wherein communicating the task to the application layer comprises integrating, into a software application, at least one of the task or the at least one task entity.
19. The computer-implemented method of claim 18 , wherein the command comprises a calendar event, the at least one task entity comprises an event date, the task comprises creating the calendar event integratable with the software application, wherein communicating the task to the application layer comprises adding the calendar event to an electronic calendar associated with the software application.