Intelligent data partitioning for distributed machine learning systems
Techniques for implementing intelligent data partitioning for a distributed machine learning (ML) system are provided. In one set of embodiments, a computer system implementing a data partition module can receive a training data instance for a ML task and identify, using a clustering algorithm, a cluster to which the training data instance belongs, the cluster being one of a plurality of clusters determined via the clustering algorithm that partition a data space of the ML task. The computer system can then transmit the training data instance to a ML worker of the distributed ML system that is assigned to the cluster, where the ML worker is configured to build or update a ML model using the training data instance.
1. A method for implementing intelligent data partitioning in a distributed machine learning (ML) system, the method comprising:
receiving, by a computer system implementing a data partition module, a training data instance for a ML task;
identifying, by the computer system using a clustering algorithm, a cluster in a plurality of data clusters to which the training data instance belongs, wherein each cluster in the plurality of data clusters corresponds to a portion of an input data space of the ML task and is assigned to an ML worker in a plurality of ML workers, and wherein the identifying comprises determining that the training data instance is more strongly associated with the cluster than other clusters in the plurality of data clusters; and
transmitting, by the computer system, the training data instance to a ML worker in the plurality of ML workers that is assigned to the cluster,
wherein the ML worker is configured to build or update a ML model using the training data instance.
2. The method of claim 1 further comprising:
receiving, by the computer system, a query data instance for the ML task;
determining, by the computer system using the clustering algorithm, that the query data instance belongs to the cluster; and
transmitting, by the computer system, the query data instance to the ML worker,
wherein the ML worker is configured to generate a decision or classification result for the query data instance using the ML model.
3. The method of claim 1 further comprising:
identifying, by the computer system, a quorum to which the training data instance belongs, the quorum comprising:
the cluster to which the training data instance belongs; and
zero or more other clusters in the plurality of data clusters, wherein clustering associations between the training data instance and the zero or more other clusters are approximately the same as a clustering association between the training data instance and the cluster to which the training data instance belongs; and
transmitting, by the computer system, the training data instance to zero or more other ML workers in the plurality of ML workers assigned to the zero or more other clusters,
wherein each of the zero or more other ML workers is configured to build or update another ML model using the training data instance.
4. The method of claim 3 further comprising:
receiving, by the computer system, a query data instance for the ML task;
determining, by the computer system, that the query data instance belongs to the quorum; and
transmitting, by the computer system, the query data instance to the ML worker and the zero or more other ML workers;
wherein each of the ML worker and the zero or more other ML workers is configured to generate a decision or classification result for the query data instance using its respective ML model.
5. The method of claim 4 wherein if the computer system receives multiple decision or classification results in response to transmitting the query data instance to the ML worker and the zero or more other ML workers, the computer system is configured to compile the multiple decision or classification results into a single, final decision or classification result.
6. The method of claim 1 wherein the plurality of data clusters includes two or more small clusters that are smaller in size than other clusters in the plurality of data clusters, and wherein the computer system is configured aggregate the two or more small clusters into a larger cluster and assign the larger cluster to a logical ML worker of the distributed ML system.
7. The method of claim 1 wherein the plurality of data clusters include a large cluster that is larger in size than other clusters in the plurality of data clusters, and wherein the computer system is configured to split the large cluster into two or more smaller clusters and assign each of the two or more smaller clusters to a separate ML worker.
8. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system implementing a data partition module for a distributed machine learning (ML) system, the program code embodying a method comprising:
receiving a training data instance for a ML task;
identifying, using a clustering algorithm, a cluster in a plurality of data clusters to which the training data instance belongs, wherein each cluster in the plurality of data clusters corresponds to a portion of an input data space of the ML task and is assigned to an ML worker in a plurality of ML workers, and wherein the identifying comprises determining that the training data instance is more strongly associated with the cluster than other clusters in the plurality of data clusters; and
transmitting the training data instance to a ML worker in the plurality of ML workers that is assigned to the cluster,
wherein the ML worker is configured to build or update a ML model using the training data instance.
9. The non-transitory computer readable storage medium of claim 8 wherein the method further comprises:
receiving a query data instance for the ML task;
determining, using the clustering algorithm, that the query data instance belongs to the cluster; and
transmitting the query data instance to the ML worker,
wherein the ML worker is configured to generate a decision or classification result for the query data instance using the ML model.
10. The non-transitory computer readable storage medium of claim 8 wherein the method further comprises:
identifying a quorum to which the training data instance belongs, the quorum comprising:
the cluster to which the training data instance belongs; and
zero or more other clusters in the plurality of data clusters, wherein clustering associations between the training data instance and the zero or more other clusters are approximately the same as a clustering association between the training data instance and the cluster to which the training data instance belongs; and
transmitting the training data instance to zero or more other ML workers in the plurality of ML workers assigned to the zero or more other clusters,
wherein each of the zero or more other ML workers is configured to build or update another ML model using the training data instance.
11. The non-transitory computer readable storage medium of claim 10 wherein the method further comprises:
receiving a query data instance for the ML task;
determining that the query data instance belongs to the quorum; and
transmitting the query data instance to the ML worker and the zero or more other ML workers;
wherein each of the ML worker and the zero or more other ML workers is configured to generate a decision or classification result for the query data instance using its respective ML model.
12. The non-transitory computer readable storage medium of claim 11 wherein if the computer system receives multiple decision or classification results in response to transmitting the query data instance to the ML worker and the zero or more other ML workers, the computer system is configured to compile the multiple decision or classification results into a single, final decision or classification result.
13. The non-transitory computer readable storage medium of claim 8 wherein the plurality of data clusters includes two or more small clusters that are smaller in size than other clusters in the plurality of data clusters, and wherein the computer system is configured aggregate the two or more small clusters into a larger cluster and assign the larger cluster to a logical ML worker of the distributed ML system.
14. The non-transitory computer readable storage medium of claim 8 wherein the plurality of data clusters include a large cluster that is larger in size than other clusters in the plurality of data clusters, and wherein the computer system is configured to split the large cluster into two or more smaller clusters and assign each of the two or more smaller clusters to a separate ML worker.
15. A computer system implementing a data partition module for a distributed machine learning (ML) system, the computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
receive a training data instance for a ML task;
identify, using a clustering algorithm, a cluster in a plurality of data clusters to which the training data instance belongs, wherein each cluster in the plurality of data clusters corresponds to a portion of an input data space of the ML task and is assigned to an ML worker in a plurality of ML workers, and wherein the identifying comprises determining that the training data instance is more strongly associated with the cluster than other clusters in the plurality of data clusters; and
transmit the training data instance to a ML worker in the plurality of ML workers that is assigned to the cluster,
wherein the ML worker is configured to build or update a ML model using the training data instance.
16. The computer system of claim 15 wherein the program code further causes the processor to:
receive a query data instance for the ML task;
determine, using the clustering algorithm, that the query data instance belongs to the cluster; and
transmit the query data instance to the ML worker,
wherein the ML worker is configured to generate a decision or classification result for the query data instance using the ML model.
17. The computer system of claim 15 wherein the program code further causes the processor to:
identify a quorum to which the training data instance belongs, the quorum comprising:
the cluster to which the training data instance belongs; and
zero or more other clusters in the plurality of data clusters, wherein clustering associations between the training data instance and the zero or more other clusters are approximately the same as a clustering association between the training data instance and the cluster to which the training data instance belongs; and
transmit the training data instance to zero or more other ML workers in the plurality of ML workers assigned to the zero or more other clusters,
wherein each of the zero or more other ML workers is configured to build or update another ML model using the training data instance.
18. The computer system of claim 17 wherein the program code further causes the processor to:
receive a query data instance for the ML task;
determine that the query data instance belongs to the quorum; and
transmit the query data instance to the ML worker and the zero or more other ML workers;
wherein each of the ML worker and the zero or more other ML workers is configured to generate a decision or classification result for the query data instance using its respective ML model.
19. The computer system of claim 18 wherein if the computer system receives multiple decision or classification results in response to transmitting the query data instance to the ML worker and the zero or more other ML workers, the computer system is configured to compile the multiple decision or classification results into a single, final decision or classification result.
20. The computer system of claim 15 wherein the plurality of data clusters includes two or more small clusters that are smaller in size than other clusters in the plurality of data clusters, and wherein the computer system is configured aggregate the two or more small clusters into a larger cluster and assign the larger cluster to a logical ML worker of the distributed ML system.
21. The computer system of claim 15 wherein the plurality of data clusters include a large cluster that is larger in size than other clusters in the plurality of data clusters, and wherein the computer system is configured to split the large cluster into two or more smaller clusters and assign each of the two or more smaller clusters to a separate ML worker.