Method and system for distributed machine learning
The present teaching relates to estimating one or more parameters on a system including a plurality of nodes. In one example, the system comprises: one or more learner nodes, each of which is configured for generating information related to a group of words for estimating the one or more parameters associated with a machine learning model; and a plurality of server nodes, each of which is configured for obtaining a plurality of sub-vectors each of which is a portion of a vector that represents a word in the group of words, updating the sub-vectors based at least partially on the information to generate a plurality of updated sub-vectors, and estimating at least one of the one or more parameters associated with the machine learning model based on the plurality of updated sub-vectors.
1. A system including a plurality of nodes, each of which has at least one processor, storage, and a communication platform connected to a network for training a machine learning model, the system comprising:
a plurality of server nodes, each of which is communicatively coupled with a plurality of learner nodes; and
a driver node configured for
receiving a request for training a model,
generating a plurality of sub-vectors, each of which is a portion of a feature vector associated with a word in a group of words,
allocating different sub-vectors associated with the word to different server nodes,
partitioning training data into a plurality of training data subsets, each of which is allocated to a learner node,
receiving a notification indicating completion of training the model, and
transmitting the model in response to the request.
2. The system of claim 1 , wherein each of the plurality of learner nodes is configured for generating information related to a group of words, wherein the information is indicative of a center word and relationship thereof with other words in the group of words.
3. The system of claim 2 , wherein the information related to the group of words includes a first index for the center word, and a second set of one or more indices for one or more positive context words of the center word.
4. The system of claim 3 , wherein:
the information related to the group of words further includes a random seed; and
each of the plurality of server nodes is further configured for generating, based on the random seed and the first index, a third set of one or more indices for one or more negative words that are not semantically related to the center word.
5. The system of claim 1 , wherein the driver node is further configured for:
selecting, based on the request, the one or more learner nodes and the plurality of server nodes for training the model; and
initializing at least some feature vectors in the plurality of feature vectors with random numbers.
6. The system of claim 1 , wherein each of the plurality of server nodes is further configured for:
calculating a plurality of partial dot products each of which is a dot product of two sub-vectors; and
providing the plurality of partial dot products to the plurality of learner nodes.
7. The system of claim 1 , wherein:
the model is utilized to determine one or more advertisements based on a query; and
the one or more advertisements are to be provided in response to the query.
8. A method implemented on at least one computing device each of which has at least one processor, storage, and a communication platform connected to a network for training a machine learning model, the method comprising:
receiving a request for training a model;
generating a plurality of sub-vectors, each of which is a portion of a feature vector associated with a word in a group of words;
allocating different sub-vectors associated with the word to different server nodes of a plurality of server nodes;
partitioning training data into a plurality of training data subsets, each of which is allocated to a learner node of a plurality of learner nodes, wherein each server node is communicatively coupled with the plurality of learner nodes;
receiving a notification indicating completion of training the model; and
transmitting the model in response to the request.
9. The method of claim 8 , wherein each of the plurality of learner nodes is configured for generating information related to a group of words, wherein the information is indicative of a center word and relationship thereof with other words in the group of words.
10. The method of claim 9 , wherein the information related to the group of words includes a first index for the center word, and a second set of one or more indices for one or more positive context words of the center word.
11. The method of claim 10 , wherein:
the information related to the group of words further includes a random seed; and
each of the plurality of server nodes is further configured for generating, based on the random seed and the first index, a third set of one or more indices for one or more negative words that are not semantically related to the center word.
12. The method of claim 8 , further comprising:
selecting, based on the request, the one or more learner nodes and the plurality of server nodes for training the model; and
initializing at least some feature vectors in the plurality of feature vectors with random numbers.
13. The method of claim 8 , further comprising:
calculating a plurality of partial dot products each of which is a dot product of two sub-vectors; and
providing the plurality of partial dot products to the plurality of learner nodes.
14. The method of claim 8 , wherein:
the model is utilized to determine one or more advertisements based on a query; and
the one or more advertisements are to be provided in response to the query.
15. A machine-readable tangible and non-transitory medium having information for training a machine learning model, wherein the information, when read by the machine, causes the machine to perform the following:
receiving a request for training a model;
generating a plurality of sub-vectors, each of which is a portion of a feature vector associated with a word in a group of words;
allocating different sub-vectors associated with the word to different server nodes of a plurality of server nodes;
partitioning training data into a plurality of training data subsets, each of which is allocated to a learner node of a plurality of learner nodes, wherein each server node is communicatively coupled with the plurality of learner nodes;
receiving a notification indicating completion of training the model; and
transmitting the model in response to the request.
16. The medium of claim 15 , wherein each of the plurality of learner nodes is configured for generating information related to a group of words, wherein the information is indicative of a center word and relationship thereof with other words in the group of words.
17. The medium of claim 16 , wherein the information related to the group of words includes a first index for the center word, and a second set of one or more indices for one or more positive context words of the center word.
18. The medium of claim 17 , wherein:
the information related to the group of words further includes a random seed; and
each of the plurality of server nodes is further configured for generating, based on the random seed and the first index, a third set of one or more indices for one or more negative words that are not semantically related to the center word.
19. The medium of claim 15 , further comprising:
selecting, based on the request, the one or more learner nodes and the plurality of server nodes for training the model; and
initializing at least some feature vectors in the plurality of feature vectors with random numbers.
20. The medium of claim 15 , wherein:
the model is utilized to determine one or more advertisements based on a query; and
the one or more advertisements are to be provided in response to the query.