AUTOMATION OF ANNOTATION FOR RECORDED WORK SESSIONS
Recorded sessions of skilled personnel at work are used to train a first machine learning (ML) tool to extract workflow maps. These maps are used to train a second ML tool to emulate at least one workflow. The first ML tool is trained to predict an annotator's output. Either ML tool can be a copilot having a microservice network architecture. Further, complex sets of workflows can be subdivided for efficient support with small ML tools for (1) workflow map extraction and (2) emulation. Based on segment demarcation accompanying a recording, segments are assigned to specialized ML extraction tools for workflow map extraction. Extracted workflow maps are used to train workflow emulator(s). Similarly, specialized ML emulation tools can emulate respective tasks. A distribution microservice can identify a task to be emulated and can invoke the appropriate specialized ML tool. Similar microservice network architectures support both applications.
1 . A method, comprising
training a machine learning (ML) tool to predict an annotator's output for one or more recorded sessions of skilled personnel at work; and
deploying one or more instances of the trained ML tool to generate respective annotations for additional recorded work sessions of additional skilled personnel at work, wherein each of the generated annotations comprises: one or more workflow maps each describing a job function performed by one or more of the additional skilled personnel.
2 . (canceled)
3 . The method of claim 2 , wherein the ML tool is a first ML tool, and the method further comprises:
collecting the generated annotations in a training corpus; and
training a second ML tool, using at least portions of the collected annotations, to emulate at least some workflows described by the workflow maps.
4 . The method of claim 3 , wherein training the second ML tool uses first portions of the collected annotations, the emulated workflows are first workflows, and the method further comprises:
training a third ML tool, using second portions of the collected annotations, to emulate second workflows described by the workflow maps;
wherein the second workflows are distinct from the first workflows.
5 . The method of claim 1 wherein, for a given session of the additional recorded work sessions, the generated annotation demarcates segments of the given session associated with respective tasks.
6 . The method of claim 1 , wherein the training comprises:
an unsupervised learning phase in which the annotator's output is pre-existing and accompanies a recording of a respective session, of the recorded sessions, as training data provided to the ML tool.
7 . The method of claim 1 , wherein the training comprises:
a supervised learning phase in which the annotator's output is pre-existing, is excluded from training data provided to the ML tool, and is compared with output of the ML tool to determine a loss function from which feedback is provided to update parameters of the ML tool.
8 . The method of claim 1 , wherein the training comprises:
a reinforcement learning phase in which the annotator's output predicted by the ML tool is rated by a human evaluator or a reward model to generate feedback for updating parameters of the ML tool.
9 . The method of claim 1 , wherein the training comprises a sequence of two or more training phases, wherein each pair of the training phases comprises a first phase and a second phase which are distinguished from each other by one or more of:
scope of knowledge of training data; configuration of the ML tool; training objective; or training modality.
10 . The method of claim 1 , wherein the ML tool is a copilot comprising a weakly connected network of microservices.
11 . The method of claim 1 , wherein the annotations are generated live during at least some of the additional work sessions.
12 . One or more non-transitory computer-readable media storing instructions executable by one or more hardware processors, the instructions comprising:
first instructions which, upon execution, cause a machine learning (ML) tool to be trained, over a sequence of two or more training phases, to predict an annotator's output for one or more recorded sessions of skilled personnel at work;
wherein the training phases are distinguishable according to one or more of: scope of knowledge of training data; configuration of the ML tool; training objective; or training modality; and
second instructions which, upon execution in a deployed instance of the trained ML tool, cause respective annotations to be generated for additional recorded work sessions.
13 . The one or more non-transitory computer-readable media of claim 12 , wherein each of the generated annotations comprises one or more workflow maps, wherein the ML tool is a first ML tool, and the instructions further comprise:
third instructions which, upon execution, cause the generated annotations to be stored in a training corpus; and
fourth instructions which, upon execution, cause a second ML tool to be trained, using at least first portions of the stored annotations, to emulate at least first workflows described by the workflow maps;
fifth instructions which, upon execution, cause a third ML tool to be trained, using at least second portions of the stored annotations, to emulate second workflows described by the workflow maps;
wherein the second workflows are distinct from the first workflows.
14 . The one or more non-transitory computer-readable media of claim 12 , wherein the ML tool being trained is an operation comprising a plurality of phases distinguished by one or more of:
scope of knowledge of training data; configuration of the ML tool; training objective; or training modality.
15 . The one or more non-transitory computer-readable media of claim 12 , wherein the ML tool is a copilot comprising a weakly connected network of microservices.
16 . The one or more non-transitory computer-readable media of claim 12 , wherein the second instructions, upon execution, cause the annotations to be generated live during at least some of the additional work sessions.
17 . A system, comprising:
one or more hardware processors with memory coupled thereto; and
computer-readable media storing instructions which, when executed, cause the one or more hardware processors to perform operations comprising:
training a machine learning (ML) tool to predict an annotator's output for one or more recorded sessions of skilled personnel performing work; and
by one or more deployed instances of the trained ML tool, generating respective annotations for additional recorded work sessions of additional skilled personnel performing work, wherein each of the generated annotations comprises: one or more workflow maps each describing a job function performed by one or more of the additional skilled personnel.
18 . The system of claim 17 , wherein each of the generated annotations comprises one or more workflow maps.
19 . The system of claim 17 wherein, for a given session of the additional recorded work sessions, the generated annotation demarcates segments of the given session associated with respective tasks.
20 . The system of claim 17 , wherein the ML tool is a copilot comprising a weakly connected network of microservices.