Distributed artificial intelligence system using transmission of compressed gradients and model parameter, and learning apparatus and method therefor

1 . A method for learning of distributed artificial intelligence by a local device, the method comprising:

updating, by the local device, parameters of a corresponding local model by training the local model based on a corresponding local dataset;

generating, by the local device, compressed gradient information by compressing gradients of the parameter of the local model;

transmitting, by the local device, the compressed gradient information to a parameter server;

reconstructing, by a parameter server, gradients of the parameters of the local model based on the compressed gradient information received from the local device; and

updating, by the parameter server, parameters of a global model based on the reconstructed gradients of the parameters of the local model,

wherein the generating comprises:

performing a block sparsification by transforming a high-dimensional local source vector which forms the gradients of the parameters of the local model, into a plurality of low-dimensional S-sparse sub-vectors; and

performing a digital encoding of transforming the plurality of low-dimensional S-sparse sub-vectors into a digital bit sequence form, and

wherein the performing the digital encoding comprises:

transforming the plurality of low-dimensional S-sparse sub-vectors into a plurality of S-dimensional linear transformed vectors; and

performing a scalar quantization of transforming each element of the plurality of S-dimensional linear transformed vectors into a digital form.

2 . The method of claim 1 , wherein the performing the block sparsification comprises:

dividing the high-dimensional local source vector into L sub-vectors; and

generating the low-dimensional S-sparse sub-vectors by selecting S largest elements from among the L sub-vectors and removing residual elements.

3 . The method of claim 2 , wherein the generating the low-dimensional S-sparse sub-vectors is performed in parallel for the L sub-vectors.

4 . The method of claim 2 , wherein the generating the low-dimensional S-sparse sub-vectors is performed based on an optimal sparsity level S associated with a quantization level Q that maximizes an optimization objective function in which local vector reconstruction performance is taken into consideration.

5 . The method of claim 1 , wherein the performing the block sparsification comprises:

reconstructing compressed gradient information digitally encoded through the digital encoding step;

storing the residual elements excluding the reconstructed gradient information from the high dimensional local source vectors; and

adding the stored residual elements to a high-dimensional source gradient vector of a next order.

6 . The method of claim 1 , wherein the transforming the plurality of low-dimensional S-sparse sub-vectors into the plurality of S-dimensional linear transformed vectors further comprises:

generating S-dimensional vectors by selecting non-zero elements from each of the plurality of low-dimensional S-sparce sub-vectors;

calculating experiential mean value and variance value of the S-dimensional vectors;

transforming the S-dimensional vectors into normalized vectors using the experiential mean value and variance value; and

transforming the normalized vectors into linear transformed vectors using S-dimensional orthogonal matrix or equivalent matrix thereof.

7 . The method of claim 6 , wherein the S-dimensional orthogonal matrix is Hadamard matrix or random Gaussian matrix.

8 . The method of claim 1 , wherein the scalar quantization is performed based on a quantization level Q that maximizes an optimization objective function in which source vector reconstruction performance is taken into consideration.

9 . The method of claim 8 , wherein the quantization level Q is determined based on allowed communication overhead.

10 . The method of claim 1 , wherein the digital encoding further comprises performing a position encoding of representing positions, at which S non-zero values are present within the plurality of low-dimensional S-sparse sub-vectors, by a digital bit sequence.

11 . The method of claim 1 , wherein the reconstructing comprises acquiring an estimated value of a normalized vector from a quantized value vector using minimized mean square error estimation scheme.

12 . The method of claim 11 , wherein the reconstructing further comprises performing an inverse normalization on the estimated value of the normalized vector using experiential mean value and variance value obtained during a normalization procedure.

13 . A method for learning of distributed artificial intelligence by a parameter server and local devices, the method comprising:

generating, by the parameter server, compressed global model parameter information by compressing parameters of a global model;

transmitting, by the parameter server, the compressed global model parameter information to the local devices;

reconstructing, by each of the local devices, the parameters of the global model based on the compressed global model parameter information received from the parameter server; and

storing, by each of the local devices, the reconstructed parameters of the global model as a local model,

wherein the generating comprises:

performing a block sparsification by transforming a high-dimensional global source vector, which forms the gradients of the parameters of the global model, into a plurality of low-dimensional S-sparse sub-vectors; and

performing a digital encoding of transforming the plurality of low-dimensional S-sparse sub-vectors into a digital bit sequence form, and

wherein the performing the digital encoding comprises:

transforming the plurality of low-dimensional S-sparse sub-vectors into a plurality of S-dimensional linear transformed vectors; and

performing a scalar quantization of transforming each element of the plurality of S-dimensional linear transformed vectors into a digital form.

14 . An apparatus for learning of distributed artificial intelligence, the apparatus comprising:

memory configured to store one or more instructions; and

a processor configured to execute the one or more instructions;

wherein, by executing the one or more instructions, the processor is further configured to:

update parameters of a local model by training the local model based on a local dataset;

generate compressed gradient information by compressing gradients of the parameters of the local model; and

control a local device to transmit the compressed gradient information to a parameter server; and

receive parameters of a global model from the parameter server,

wherein the processor, for the generating the compressed gradient information, is further configured to:

transform a high-dimensional local source vector which forms the gradients of the parameters of the local model, into a plurality of low-dimensional S-sparse sub-vectors through a block sparsification process; and

transform the plurality of low-dimensional S-sparse sub-vectors into a digital bit sequence form through a digital encoding process, by:

transforming the plurality of low-dimensional S-sparse sub-vectors into a plurality of S-dimensional linear transformed vectors; and

performing a scalar quantization of transforming each element of the plurality of S-dimensional linear transformed vectors into a digital form.

15 . The apparatus of claim 14 , wherein the parameters of the global model received from the parameter server is compressed information of the parameters of the global model.

16 . The apparatus of claim 14 , wherein the processor is further configured to:

update the parameters of the local model based on the parameters of the global model; and

update the parameters of the local model again by training the local model based on a new local dataset.