Multimodal retrieval and execution monitoring using rich recipe representation
The disclosure deals with a system and method for improved representation and retrieval of recipes or workflows. Recipes or workflows such as for preparing food or assembling furniture or performing other complex activities exist as textual or image documents, which makes it difficult for machines to read, reason, and handle ambiguity. The present disclosure provides a Rich Recipe Representation (“R3”), which is enhanced with additional knowledge such as outcomes like allergen information, possible failures, and solutions for each atomic step (such as a cooking step). The disclosed R3 is used in a web-based decision support system that helps users perform constrained queries using multiple modalities while also monitoring execution of an agent cooking or otherwise acting based on it.
1. Methodology for multimodal recipe retrieval and execution monitoring, comprising:
providing a user interface;
maintaining a database comprising a knowledge graph of external information viewable by a user via the user interface;
maintaining a searchable repository of rich recipe representations, each of which comprise a recipe and associated ingredients in a unified representation of instructions, with each instruction recorded in rich recipe representation with one or more atomic actions corresponding to an executable action, along with its description in text and images, condition for successful execution, and pointer to pertinent data in the knowledge graph of external information;
making a recipe query input by the user via the user interface to the searchable repository in the form of text, image or a combination thereof;
notifying the user via the user interface of recipe query output obtained from the searchable repository in the form of text, image or a combination thereof;
executing the recipe instructions; and
monitoring execution of the recipe atomic actions as established by the corresponding conditions retrieved from the recipe query;
wherein the knowledge graph of external information includes information about possible failures and solutions for each atomic action; and
the methodology further comprises notifying the user via the user interface whenever monitoring execution of the recipe atomic actions indicates occurrence of a failure condition for a specific atomic action, and comprises providing the user possible solutions via the external information via the user interface.
2. Methodology in accordance with claim 1 , wherein:
the knowledge graph of external information includes at least one of information about allergens in ingredients and about condition of food prepared; and
the recipe query input includes a constraint on outcome corresponding with at least information in the knowledge graph of external information.
3. Methodology in accordance with claim 1 , wherein the recipe query input includes a constraint on outcome corresponding with at least the type and the order of actions according to instructions for a recipe.
4. Methodology in accordance with claim 1 , wherein the unified representation of instructions for rich recipe representations (R3) comprises a machine processable JavaScript Object Notation (JSON) representation which comprises an open standard, human-readable text-based format for representing structured data.
5. Methodology in accordance with claim 1 , further comprising using a web-based user interface for use in conducting a recipe query input to the searchable repository.
6. Methodology in accordance with claim 1 , wherein the recipe query input is one of a constrained query and an expressive query.
7. Methodology in accordance with claim 1 , wherein the knowledge graph of external information includes information about allergen information.
8. Methodology for multimodal retrieval and execution monitoring for a workflow for a user accomplishing a task, comprising:
providing a user interface;
maintaining a database comprising a knowledge graph of external information viewable by a user via the user interface;
maintaining a searchable repository of rich workflow representations, each of which comprise a workflow and associated materials in a unified representation of instructions, with each instruction recorded in rich workflow representation with one or more atomic actions corresponding to an executable action, along with its description in text and images, condition for successful execution, and pointer to pertinent data in the knowledge graph of external information;
making a workflow query input by the user via the user interface to the searchable repository in the form of text, image or a combination thereof;
notifying the user via the user interface of workflow query output obtained from the searchable repository in the form of text, image or a combination thereof;
executing the workflow instructions; and
monitoring execution of the workflow atomic actions as established by the corresponding conditions retrieved from the workflow query;
wherein the knowledge graph of external information includes information about possible failures and solutions for each atomic action; and
the methodology further comprises notifying the user via the user interface whenever monitoring execution of the workflow atomic actions indicates occurrence of a failure condition for a specific atomic action, and comprises providing the user possible solutions via the external information via the user interface.
9. Methodology in accordance with claim 8 , wherein the workflow for a user accomplishing a task includes at least one of a recipe for preparing food, a set of instructions for assembling a product, and a set of instructions for performing complex activities.
10. Methodology in accordance with claim 8 , wherein:
the knowledge graph of external information includes at least one of information about materials; and
the workflow query input includes a constraint on outcome corresponding with at least information in the knowledge graph of external information.
11. Methodology in accordance with claim 8 , wherein the workflow query input includes a constraint on outcome corresponding with at least the type and the order of actions according to instructions for a workflow.
12. Methodology in accordance with claim 8 , wherein the unified representation of instructions for rich workflow representations comprises a machine processable JavaScript Object Notation (JSON) representation which comprises an open standard, human-readable text-based format for representing structured data.
13. Methodology in accordance with claim 8 , further comprising using a web-based user interface for conducting a workflow query input to the searchable repository.
14. Methodology in accordance with claim 8 , wherein the workflow query input is a constrained query.
15. Methodology in accordance with claim 8 , wherein the knowledge graph of external information includes information about materials for each atomic action.
16. A system for multimodal recipe retrieval and execution monitoring, comprising:
a user interface;
a database comprising a knowledge graph of external information viewable by a user via the user interface and including information about possible failures and solutions for each atomic action;
a searchable repository of rich recipe representations, each of which comprise a recipe and associated ingredients in a unified representation of instructions, with each instruction recorded in rich recipe representation with one or more atomic actions corresponding to an executable action, along with its description in text and images, condition for successful execution, and pointer to pertinent data in the knowledge graph of external information; and
one or more processors programmed for
making a recipe query input by the user via the user interface to the searchable repository in the form of text, image or a combination thereof;
notifying the user via the user interface of recipe query output obtained from the searchable repository in the form of text, image or a combination thereof;
monitoring execution of the recipe atomic actions by a user as established by the corresponding conditions retrieved from the recipe query; and
notifying the user via the user interface whenever monitoring execution of the recipe atomic actions indicates occurrence of a failure condition for a specific atomic action, and providing the user possible solutions via the external information via the user interface.
17. A system in accordance with claim 16 , wherein:
the knowledge graph of external information includes at least one of information about allergens in ingredients and about condition of food prepared; and
the recipe query input includes a constraint on outcome corresponding with at least information in the knowledge graph of external information.
18. A system in accordance with claim 16 , wherein the recipe query input includes a constraint on outcome corresponding with at least the type of action to be taken when following instructions for a recipe, and the order of actions to be taken when following instructions for a recipe.
19. A system in accordance with claim 16 , wherein the unified representation of instructions for rich recipe representations (R3) comprises a machine processable JavaScript Object Notation (JSON) representation which comprises an open standard, human-readable text-based format for representing structured data.
20. A system in accordance with claim 16 , further comprising using a web-based user interface for use in conducting a recipe query input to the searchable repository.
21. A system in accordance with claim 16 , wherein the recipe query input is one of a constrained query and an expressive query.
22. A system in accordance with claim 16 , wherein the knowledge graph of external information includes information about allergen information.
23. A system in accordance with claim 16 , wherein said database and said searchable repository each comprise one or more network-based non-transitory storage devices.
24. A system in accordance with claim 23 , wherein said one or more processors are further programmed for periodically updating at least one of said database and said searchable repository.
25. A system in accordance with claim 24 , wherein said one or more processors further comprise an AI-based system using analytical/optimization techniques.
26. A system in accordance with claim 16 , wherein said system comprises one of a web-based application and a stand-alone application running on a personal computer.