Apparatus and method for distributed model training, device, and computer readable storage medium

1 . An apparatus for distributed training of a model, the apparatus comprising a distributed reader, a distributed trainer, and a distributed parameter server that are mutually independent, wherein:

a reader in the distributed reader is configured to acquire a training sample, and load the acquired training sample to a corresponding trainer in the distributed trainer;

the trainer in the distributed trainer is configured to perform model training based on the loaded training sample to obtain gradient information, wherein the distributed trainer comprises a plurality of trainers, and in a model training process, computing powers of the plurality of trainers match loads of the plurality of trainers by adjusting the computing powers of the plurality of trainers based on a load balancing strategy, wherein the plurality of trainers are all in a status of full load by matching the computing powers of the plurality of trainers;

a parameter server in the distributed parameter server is configured to update a parameter of an initial model based on the gradient information of the distributed trainer to obtain a trained target model; and

the apparatus further comprises a communicator unit configured to perform data interaction on parameters and gradients between the distributed trainer and the distributed parameter server, wherein the communicator unit is further configured to perform global shuffle of the training sample in the distributed trainer.

2 . The apparatus according to claim 1 , further comprising a distributed parameter processor, wherein:

a parameter processor in the distributed parameter processor is configured to send, receive, and aggregate a parameter and a gradient in the distributed trainer.

3 . The apparatus according to claim 1 , wherein the reader in the distributed reader and the corresponding trainer in the distributed trainer perform data interaction in a multi-threaded fully asynchronous approach.

4 . The apparatus according to claim 3 , wherein each reading thread in the reader in the distributed reader acquires a training sample respectively, and loads the acquired training sample to an information queue corresponding to each reading thread, so that each training thread of the trainer corresponding to the reader acquires the training sample from the corresponding information queue.

5 . The apparatus according to claim 1 , wherein:

parameters of the initial model are divided into a target number of sets of parameters based on a preset method, wherein the target number represents a number of parameter servers in the distributed parameter server; and

each parameter server in the distributed parameter server is configured to store a respective set of parameters of the target number of sets of parameters.

6 . The apparatus according to claim 5 , wherein:

the trainer in the distributed trainer is further configured to determine a target parameter server in the distributed parameter server that performs data interaction with the trainer through a parameter identifier; and

the communicator unit is further configured to perform data interaction between the trainer in the distributed trainer and the target parameter server corresponding to the trainer.

7 . The apparatus according to claim 5 , wherein each parameter is associated with a corresponding parameter identifier represented by a number, and the parameters of the initial model are divided into the target number of sets of parameters by performing a modulo operation on the corresponding parameter identifier based on a value represented by the target number.

8 . The apparatus according to claim 1 , further comprising a non-transitory machine-readable medium,

wherein the non-transitory machine-readable medium stores instructions executable to provide an interface for interacting with a user.

9 . A method for distributed training of a model, the method comprising:

dividing a training process of an initial model into a plurality of distributed functional modules, and determining a communication interface between the plurality of distributed functional modules, wherein the plurality of distributed functional modules are independent of each other, and comprise a distributed reader, a distributed trainer and a distributed parameter server;

acquiring, by a reader in the distributed reader, a training sample, and loading the acquired training sample to a corresponding trainer in the distributed trainer; and

performing, by the trainer in the distributed trainer, model training based on the loaded training sample, and obtaining a trained target model based on data interaction between the distributed trainer and the distributed parameter server, wherein the distributed trainer comprises a plurality of trainers, and in a model training process, computing powers of the plurality of trainers match loads of the plurality of trainers by adjusting the computing powers of the trainers based on a load balancing strategy, wherein the plurality of trainers are all in a status of full load by matching the computing powers of the plurality of trainers and

wherein the plurality of distributed functional modules comprises a communicator unit, and performing the model training comprises performing, by the communicator unit, data interaction on parameters and gradients between the distributed trainer and the distributed parameter server, and performing, by the communicator unit, global shuffle of the training sample in the distributed trainer.

10 . The method according to claim 9 , wherein:

the plurality of distributed functional modules further comprises a distributed parameter processor; and

the performing, by the trainer in the distributed trainer, model training based on the loaded training sample, and obtaining a trained target model based on data interaction between the distributed trainer and the distributed parameter server, comprise:

sending, receiving, and aggregating a parameter and a gradient in the distributed trainer through a parameter processor in the distributed parameter processor, in a model training process of the trainer in the distributed trainer.

11 . The method according to claim 9 , wherein the reader in the distributed reader and the corresponding trainer in the distributed trainer perform data interaction in a multi-threaded fully asynchronous approach.

12 . The method according to claim 11 , wherein the acquiring, by a reader in the distributed reader, a training sample, and loading the acquired training sample to a corresponding trainer in the distributed trainer comprise:

for each reader in the distributed reader, acquiring, by each reading thread in the reader, a training sample respectively, and loading the acquired training sample to an information queue corresponding to each reading thread, so that each training thread of the trainer corresponding to the reader acquires the training sample from the corresponding information queue.

13 . The method according to claim 9 , further comprising:

dividing parameters of the initial model into a target number of sets of parameters based on a preset method, wherein the target number represents a number of parameter servers in the distributed parameter server; and

storing, by each parameter server in the distributed parameter server, a respective set of parameters of the target number of sets of parameters.

14 . The method according to claim 13 , wherein the obtaining a trained target model based on data interaction between the distributed trainer and the distributed parameter server comprises:

for each trainer in the distributed trainer, determining a target parameter server in the distributed parameter server that performs data interaction with the trainer through a parameter identifier; and

obtaining the trained target model, based on the data interaction between the trainer and the target parameter server corresponding to the trainer.

15 . An electronic device, comprising:

at least one processor; and

a memory communicatively connected to the at least one processor,

wherein the memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform the method according to claim 9 .

16 . A non-transitory computer readable storage medium storing computer instructions, the computer instructions being used to cause a computer to perform the method according to claim 9 .