Performing batched training for machine-learning pipelines
A computer-implemented method according to one embodiment includes identifying a machine learning pipeline and a plurality of training data batches; creating a plurality of tasks, based on the machine learning pipeline; and determining an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach.
1. A computer-implemented method, comprising:
identifying a machine learning pipeline and a plurality of training data batches;
creating a plurality of tasks, based on the machine learning pipeline, the tasks including a transform task that includes an instance where an operator takes training data of one of the training data batches as input and outputs computed data, the tasks including a partial-fit task that includes an instance where an operator is trained partially using a single batch of the plurality of training data batches; and
determining an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach, wherein the partial-fit task is prioritized over the transform task such that the partial-fit task is placed before the transform task within the determined order.
2. The computer-implemented method of claim 1 , wherein the machine learning pipeline includes a plurality of operators that perform the tasks on the training data batches, wherein each of the operators is connected to another of the operators via a data flow edge within the machine learning pipeline, the data flow edge indicating a flow of data from one of the operators to another of the operators.
3. The computer-implemented method of claim 1 , wherein each of the training data batches includes a different subset of a larger group of training data.
4. The computer-implemented method of claim 1 , wherein the tasks include operations performed on training data by operators within the machine learning pipeline.
5. The computer-implemented method of claim 1 , comprising generating a task graph from the plurality of tasks; identifying one or more of the tasks in the task graph as moot; and removing the one or more moot tasks from a schedule.
6. The computer-implemented method of claim 1 , wherein the tasks include partial-fit and transform tasks, wherein a task graph is created by analyzing tasks performed by each of a plurality of operators within the machine learning pipeline, where dependencies between the tasks are identified and incorporated into the task graph.
7. The computer-implemented method of claim 1 , wherein the plurality of tasks are included within a task graph that is created lazily by adding a beginning one of the tasks to the task graph, and adding subsequent ones of the tasks to the task graph dynamically as needed.
8. The computer-implemented method of claim 1 , wherein a schedule is created that indicates the order in which the plurality of tasks is implemented.
9. The computer-implemented method of claim 1 , wherein the order in which the plurality of tasks is implemented is based at least in part on one or more parameters selected from the group consisting of: a batch ID for each task, a pipeline topology, and a batch spilling state.
10. The computer-implemented method of claim 1 , wherein the resource usage-aware approach accounts for resource usage including data loading and spilling within the machine learning pipeline, wherein the tasks include a spill task that transfers data from local memory utilized by the machine learning pipeline to a different storage location.
11. The computer-implemented method of claim 1 , wherein in response to determining that a task graph indicates that any of a plurality of tasks may be performed in a single time period, predetermined decision criteria is applied to metadata associated with each of the tasks that may be performed in the single time period to determine an order in which the tasks that may be performed in the single time period are to be performed within a schedule.
12. The computer-implemented method of claim 1 , wherein the resource usage-aware approach removes redundant tasks within a schedule.
13. The computer-implemented method of claim 1 , wherein one or more of the training data batches is cached and used for multiple different machine learning pipelines.
14. The computer-implemented method of claim 1 , wherein multiple different machine learning pipelines are trained in parallel using the training data batches, and tasks are scheduled across the multiple different machine learning pipelines.
15. The computer-implemented method of claim 1 , wherein multiple different tasks are implemented in parallel within a schedule.
16. The computer-implemented method of claim 1 , comprising adding batching to non-batched pipeline components via sampling.
17. The computer-implemented method of claim 1 , wherein batched training is performed within a hybrid execution mode for only a subset of a plurality of operators in the machine learning pipeline.
18. A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method comprising:
identifying, by the one or more processors, a machine learning pipeline and a plurality of training data batches;
creating, by the one or more processors, a plurality of tasks, based on the machine learning pipeline, the tasks including a transform task that includes an instance where an operator takes training data of one of the training data batches as input and outputs computed data, the tasks including a partial-fit task that includes an instance where an operator is trained partially using a single batch of the plurality of training data batches; and
determining, by the one or more processors, an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach, wherein the partial-fit task is prioritized over the transform task such that the partial-fit task is placed before the transform task within the determined order.
19. A system, comprising:
a processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
identify a machine learning pipeline and a plurality of predefined training data batches;
create a plurality of tasks, based on the machine learning pipeline, the tasks including a transform task that includes an instance where an operator takes training data of one of the training data batches as input and outputs computed data, the tasks including a partial-fit task that includes an instance where an operator is trained partially using a single batch of the plurality of training data batches; and
determine an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach, wherein the partial-fit task is prioritized over the transform task such that the partial-fit task is placed before the transform task within the determined order.
20. The computer-implemented method of claim 1 , wherein the tasks include a training task that trains an operator; and comprising identifying the training task as moot in response to determining that, according to a schedule, the operator is already trained before the training task occurs within the schedule; and removing the training task from the schedule.