Granted Patent B2

US 10,664,777 · App. 14/852,397 · Granted May 26, 2020

Abstract

Claims (53)

1. A method comprising:

receiving, by one or more processors of an information processing system, workflow results from completion of a first set of a plurality of iterations of a workflow, each iteration of the workflow comprising one or more tasks performed by one or more distributed computing systems;

extracting, by the one or more processors and from the workflow results, training data for training a machine learning model to assess the workflow structure, wherein the workflow structure comprises the specification of tasks within each iteration of the workflow, wherein extracting the training data comprises extracting features from workflow results that satisfy an adjudication rule and that are appropriate for use by the machine learning model;

training, by the one or more processors, the machine learning model for assessing the workflow structure using the features of the training data extracted from the workflow results;

determining, by the one or more processors using the trained machine learning model for assessing the workflow structure, a forecast of an effect on one or more metrics of altering the workflow structure in accordance with one or more alterations proposed based on the machine learning model;

producing, by the one or more processors and based on the forecast and the machine learning model for assessing the workflow, a recommended alteration to the workflow for one or more remaining sets of the plurality of iterations of the workflow, wherein the recommended alteration to the workflow comprises a percentage of the one or more tasks of the workflow to automate;

receiving, by the one or more processors after producing the recommended alteration to the workflow, new workflow results from completion of a second set of the plurality of iterations of the workflow; and

calibrating, by the one or more processors, the machine learning model for assessing the workflow structure using features extracted from training data extracted from the new workflow results.

2. The method of claim 1 , wherein the forecast indicates that altering the workflow structure in a specified fashion will result in benefits comprising an increase in efficiency, an increase in a level of quality of the workflow results, or a decrease in a cost associated with the workflow.

3. The method of claim 2 , further comprising:

altering the workflow in accordance with the recommendation;

testing the altered workflow to obtain new workflow results; and

validating the new workflow results to determine whether the benefits were achieved.

4. The method of claim 1 , wherein the recommendation comprises identifying at least one task in the workflow structure to be split into one or more sub-tasks.

5. The method of claim 4 , wherein the recommendation comprises specifying task completion quality requirements for at least one of the sub-tasks that differ from original task completion quality requirements specified for the identified task.

6. The method of claim 1 , wherein the recommendation comprises selecting at least one task in the workflow structure to be automated.

7. The method of claim 6 , wherein the task is selected to be automated based on an attribute of the task that suggests that the task is a good candidate for automation.

8. The method of claim 7 , wherein the attribute comprises:

a number of iterations in the first set of iterations;

a number of iterations in a second set of iterations; or

a type of the task.

9. The method of claim 6 , wherein the recommendation comprises presenting a forecasted accuracy for at least one portion of the workflow structure that is automated.

10. The method of claim 1 , wherein the recommendation comprises adding a new task to the workflow structure.

11. The method of claim 1 , further comprising:

receiving a minimum acceptable accuracy level; and

providing a workflow alteration strategy for each of the tasks in the workflow structure to achieve the acceptable accuracy level.

12. The method of claim 1 , wherein the adjudication rule provides a degree of confidence in the correctness or accuracy of a workflow result.

13. One or more computer-readable non-transitory storage media embodying software comprising instructions operable when executed to:

receive results from completion of a first set of a plurality of iterations of a workflow, each iteration of the workflow comprising one or more tasks performed by one or more distributed computing systems;

extract, from the workflow results, training data for training a machine learning model to assess the workflow structure, wherein the workflow structure comprises the specification of tasks within each iteration of the workflow, wherein extracting the training data comprises extracting features from workflow results that satisfy an adjudication rule and that are appropriate for use by the machine learning model;

train the machine learning model for assessing the workflow structure using the features of the training data extracted from the workflow results;

determine, using the trained machine learning model for assessing the workflow structure, a forecast of an effect on one or more metrics of altering the workflow structure in accordance with one or more alterations proposed based on the machine learning model;

produce, based on the forecast and the machine learning model for assessing the workflow, a recommended alteration to the workflow for one or more remaining sets of the plurality of iterations of the workflow, wherein the recommended alteration to the workflow comprises a percentage of the one or more tasks of the workflow to automate;

receive, after producing the recommended alteration to the workflow, new workflow results from completion of a second set of the plurality of iterations of the workflow; and

calibrate the machine learning model for assessing the workflow structure using features extracted from training data extracted from the new workflow results.

14. The media of claim 13 , wherein the forecast indicates that altering the workflow structure in a specified fashion will result in benefits comprising an increase in efficiency, an increase in a level of quality of the workflow results, or a decrease in a cost associated with the workflow.

15. The media of claim 14 , the software further comprising instructions operable when executed to:

alter the workflow in accordance with the recommendation;

test the altered workflow to obtain new workflow results; and

validate the new workflow results to determine whether the benefits were achieved.

16. A system comprising one or more processors and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:

receive results from completion of a first set of a plurality of iterations of a workflow, each iteration of the workflow comprising one or more tasks performed by one or more distributed computing systems;

extract, from the workflow results, training data for training a machine learning model to assess the workflow structure, wherein the workflow structure comprises the specification of tasks within each iteration of the workflow, wherein extracting the training data comprises extracting features from workflow results that satisfy an adjudication rule and that are appropriate for use by the machine learning model;

train the machine learning model for assessing the workflow structure using the features of the training data extracted from the workflow results;

determine, using the trained machine learning model for assessing the workflow structure, a forecast of an effect on one or more metrics of altering the workflow structure in accordance with one or more alterations proposed based on the machine learning model;

produce, based on the forecast and the machine learning model for assessing the workflow, a recommended alteration to the workflow for one or more remaining sets of the plurality of iterations of the workflow, wherein the recommended alteration to the workflow comprises a percentage of the one or more tasks of the workflow to automate;

receive, after producing the recommended alteration to the workflow, new workflow results from completion of a second set of the plurality of iterations of the workflow; and

calibrate the machine learning model for assessing the workflow structure using features extracted from training data extracted from the new workflow results.

17. The system of claim 16 , wherein the forecast indicates that altering the workflow structure in a specified fashion will result in benefits comprising an increase in efficiency, an increase in a level of quality of the workflow results, or a decrease in a cost associated with the workflow.

18. The system of claim 17 , the processors being further operable when executing the instructions to:

alter the workflow in accordance with the recommendation;

test the altered workflow to obtain new workflow results; and

validate the new workflow results to determine whether the benefits were achieved.

Assignments (9)