Method and apparatus for decentralized supervised learning in NLP applications
A method of training a neural network as a natural language processing, NLP, model, comprises: inputting annotated training data to first architecture portions of the neural network, the first architecture portions being executed respectively in a plurality of distributed client computing devices in communication with a server computing device, the training data being derived from text data private to the client computing device in which the first architecture portion is executed, the server computing device having no access to any of the private text data; deriving from the training data, using the first architecture portions, weight matrices of numeric weights which are decoupled from the private text data; concatenating the weight matrices, in a second architecture portion of the neural network executed in the server computing device, to obtain a single concatenated weight matrix; and training, on the second architecture portion, the NLP model using the concatenated weight matrix.
1. A method of training a neural network as a natural language processing, NLP, model, the method comprising:
inputting respective sets of annotated training data to a plurality of first architecture portions of the neural network, which first architecture portions are executed in respective client computing devices of a plurality of distributed client computing devices in communication with a server computing device, wherein each set of training data is derived from a set of text data which is private to the client computing device in which the first architecture portion is executed, the server computing device having no access to any of the private text data sets, and all sets of training data share a common encoding;
deriving from the sets of annotated training data, using the first architecture portions, respective weight matrices of numeric weights which are decoupled from the private text data sets;
concatenating, in a second architecture portion of the neural network which is executed in the server computing device, the weight matrices received from the client computing devices to obtain a single concatenated weight matrix; and
training, on the second architecture portion, the NLP model using the concatenated weight matrix;
wherein the sets of training data in the common encoding are derived by pre-processing private sets of text data in respective client computing devices by:
carrying out on the set of text data in each client computing device a vocabulary codification process to ensure a common vocabulary codification amongst all the training data to be provided by the client computing devices, wherein in the vocabulary codification process a common alphanumeric character-level representation is established for the vocabulary that uses characters decided by the server computing device, and
using the predefined common character-level representations and predefined common setting parameters, carrying out in each client computing device a word embedding process in which the text data is mapped to vectors of real numbers.
2. Apparatus for training a neural network as a natural language processing, NLP, model, the apparatus comprising:
a plurality of distributed client computing devices to execute respectively a plurality of first architecture portions of the neural network, wherein each first architecture portion receives a set of annotated training data derived from a set of text data which is private to the client computing device in which the first architecture portion is executed, all sets of training data sharing a common encoding; and
a server computing device in communication with each of the client computing devices of the plurality, the server computing device to execute a second architecture portion of the neural network, the server computing device having no access to any of the private text data sets;
wherein:
the first architecture portions derive, from the sets of annotated training data, respective weight matrices of numeric weights which are decoupled from the private text data sets, and
the weight matrices received from the client computing devices are concatenated in the second architecture portion to obtain a single concatenated weight matrix, the NLP model being trained on the second architecture portion using the concatenated weight matrix; and
wherein the client computing devices are operable to derive respective sets of training data in the common encoding by pre-processing respective private sets of text data by:
carrying out on the set of text data in each client computing device a vocabulary codification process to ensure a common vocabulary codification amongst all the training data to be provided by the client computing devices, wherein in the vocabulary codification process a common alphanumeric character-level representation is established for the vocabulary that uses characters decided by the server computing device, and
using the predefined common character-level representations and predefined common setting parameters, carrying out in each client computing device a word embedding process in which the text data is mapped to vectors of real numbers.