Suggesting activities

1 . A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:

receiving a real-time image or video depicting an entity comprising at least one of a person or a system;

receiving state inputs pertinent to at least one of a user or the entity;

determining prospective instructions for the user to interact with the entity based on at least one of the state inputs;

for each prospective instruction, executing a predictive model over a time horizon in the future simulating execution of the prospective instruction to predict at least one corresponding predicted outcome for execution of the prospective instruction;

executing a plurality of evaluators, each evaluator having a corresponding objective and configured to, for each prospective instruction:

evaluate the prospective instruction based on whether the at least one corresponding predicted outcome for execution of the prospective instruction satisfies the corresponding objective of the evaluator; and

output an evaluation of the prospective instruction,

wherein at least one evaluator comprises a cognitive computing model trained to evaluate a given prospective instruction based on whether at least one corresponding predicted outcome for execution of the given prospective instruction satisfies the corresponding objective of the evaluator, and

wherein at least two evaluators have functionally distinct corresponding objectives, wherein a first evaluator is configured to evaluate prospective instructions based on a first criterion and a second evaluator is configured to evaluate prospective instructions based on a second criterion different from the first criterion;

selecting a suggested instruction from the prospective instructions based on the evaluations of the prospective instructions of one or more evaluators; and

in real-time:

augmenting the real-time image or video, by:

generating a graphical overlay illustrating the suggested instruction, the graphical overlay comprising an interactive graphical element configured to receive an input from the user in response to the suggested instruction; and

superimposing the graphical overlay onto and compositing the graphical overlay with the real-time image or video depicting the entity to create an augmented reality view; and

displaying, on a screen in communication with the data processing hardware, the augmented reality view.

2 . The computer-implemented method of claim 1 , further comprising receiving feedback on execution of the suggested instruction, wherein the predictive model learns a preference of at least one of the user or the system based on the received feedback.

3 . The computer-implemented method of claim 1 , wherein the state inputs comprise one or more of:

sensor inputs from one or more sensors in communication with the data processing hardware;

application inputs received from one or more software applications executing on the data processing hardware or a remote device in communication with the data processing hardware; or

user inputs received from a graphical user interface of a user of the system.

4 . The computer-implemented method of claim 1 , wherein at least one evaluator elects to participate or not participate in evaluating the prospective instructions based on at least one state input.

5 . The computer-implemented method of claim 1 , wherein each evaluator is configured to:

determine whether any state input is of an input type associated with the evaluator; and

for each state input that is of an input type associated with the evaluator, incrementing an influence value associated with the evaluator,

wherein when the influence value of the evaluator satisfies an influence value criteria, the evaluator participates in evaluating the prospective instructions, and when the influence value of the evaluator does not satisfy the influence value criteria, the evaluator does not participate in evaluating the prospective instructions.

6 . The computer-implemented method of claim 5 , wherein the evaluation of at least one evaluator is weighted based on the corresponding influence value of the at least one evaluator.

7 . The computer-implemented method of claim 1 , wherein at least one evaluator evaluates the prospective instructions based on a history of previously selected suggested instructions.

8 . The computer-implemented method of claim 1 , wherein a first evaluator evaluates the prospective instructions based on an evaluation by a second evaluator of the prospective instructions.

9 . A computing system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving a real-time image or video depicting an entity comprising at least one of a person or a system;

receiving state inputs pertinent to at least one of a user or the entity;

determining prospective instructions for the user to interact with the entity based on at least one of the state inputs;

for each prospective instruction, executing a predictive model over a time horizon in the future simulating execution of the prospective instruction to predict at least one corresponding predicted outcome for execution of the prospective instruction;

executing a plurality of evaluators, each evaluator having a corresponding objective and configured to, for each prospective instruction:

evaluate the prospective instruction based on whether the at least one corresponding predicted outcome for execution of the prospective instruction satisfies the corresponding objective of the evaluator; and

output an evaluation of the prospective instruction,

wherein at least one evaluator comprises a cognitive computing model trained to evaluate a given prospective instruction based on whether at least one corresponding predicted outcome for execution of the given prospective instruction satisfies the corresponding objective of the evaluator, and

wherein at least two evaluators have functionally distinct corresponding objectives, wherein a first evaluator is configured to evaluate prospective instructions based on a first criterion and a second evaluator is configured to evaluate prospective instructions based on a second criterion different from the first criterion;

selecting a suggested instruction from the prospective instructions based on the evaluations of the prospective instructions of one or more evaluators; and

in real-time:

augmenting the real-time image or video, by:

generating a graphical overlay illustrating the suggested instruction, the graphical overlay comprising an interactive graphical element configured to receive an input from the user in response to the suggested instruction; and

superimposing the graphical overlay onto and compositing the graphical overlay with the real-time image or video depicting the entity to create an augmented reality view; and

displaying, on a screen in communication with the data processing hardware, the augmented reality view.

10 . The computing system of claim 9 , further comprising receiving feedback on execution of the suggested instruction, wherein the predictive model learns a preference of at least one of the user or the system based on the received feedback.

11 . The computing system of claim 9 , wherein the state inputs comprise one or more of:

sensor inputs from one or more sensors in communication with the data processing hardware;

application inputs received from one or more software applications executing on the data processing hardware or a remote device in communication with the data processing hardware; or

user inputs received from a graphical user interface of a user of the system.

12 . The computing system of claim 9 , wherein at least one evaluator elects to participate or not participate in evaluating the prospective instructions based on at least one state input.

13 . The computing system of claim 9 , wherein each evaluator is configured to:

determine whether any state input is of an input type associated with the evaluator; and

for each state input that is of an input type associated with the evaluator, incrementing an influence value associated with the evaluator,

wherein when the influence value of the evaluator satisfies an influence value criteria, the evaluator participates in evaluating the prospective instructions, and when the influence value of the evaluator does not satisfy the influence value criteria, the evaluator does not participate in evaluating the prospective instructions.

14 . The computing system of claim 13 , wherein the evaluation of at least one evaluator is weighted based on the corresponding influence value of the at least one evaluator.

15 . The computing system of claim 9 , wherein at least one evaluator evaluates the prospective instructions based on a history of previously selected suggested instructions.

16 . The computing system of claim 9 , wherein a first evaluator evaluates the prospective instructions based on an evaluation by a second evaluator of the prospective instructions.