Systems and Methods of Training Processing Engines

1 . A computer-implemented method of training processing engines, the method including:

accessing a plurality of processing engines,

wherein each processing engine in the plurality of processing engines has at least a first processing module and a second processing module,

wherein the first processing module in each processing engine is different from a corresponding first processing module in every other processing engine, and

wherein the second processing module in each processing engine is same as a corresponding second processing module in every other processing engine;

deploying each processing engine to a respective hardware module in a plurality of hardware modules for training;

processing, during forward pass stage of the training, inputs through the first processing modules of the processing engines and producing an intermediate output for each first processing module;

combining, during the forward pass stage of the training, intermediate outputs across the first processing modules and producing a combined intermediate output for each first processing module;

processing, during the forward pass stage of the training, combined intermediate outputs through the second processing modules of the processing engines and producing a final output for each second processing module;

determining, during backward pass stage of the training, gradients for each second processing module based on corresponding final outputs and corresponding ground truths;

accumulating, during the backward pass stage of the training, the gradients across the second processing modules and producing accumulated gradients; and

updating, during the backward pass stage of the training, weights of the second processing modules based on the accumulated gradients and producing updated second processing modules.

2 . The computer-implemented method of claim 1 , further including:

determining, during the backward pass stage of the training, gradients for each first processing module based on the combined intermediate outputs, the corresponding final outputs, and the corresponding ground truths; and

updating, during the backward pass stage of the training, weights of the first processing modules based on the determined gradients and producing updated first processing modules.

3 . The computer-implemented method of claim 2 , further including:

storing the updated first processing modules and the updated second processing modules as updated processing engines; and

making the updated processing engines available for inference.

4 . The computer-implemented method of claim 1 , wherein the hardware module is a computing device and/or edge device.

5 . The computer-implemented method of claim 1 , wherein the hardware module is a chip.

6 . The computer-implemented method of claim 1 , wherein the hardware module is a part of a chip.

7 . The computer-implemented method of claim 1 , further including accumulating the gradients across the second processing modules and producing the accumulated gradients by determining weighted averages of the gradients.

8 . The computer-implemented method of claim 1 , further including accumulating the gradients across the second processing modules and producing the accumulated gradients by determining averages of the gradients.

9 . The computer-implemented method of claim 1 , further including combining the intermediate outputs across the first processing modules and producing the combined intermediate output for each first processing module by concatenating the intermediate outputs across the first processing modules.

10 . The computer-implemented method of claim 1 , further including combining the intermediate outputs across the first processing modules and producing the combined intermediate output for each first processing module by summing the intermediate outputs across the first processing modules.

11 . The computer-implemented method of claim 1 , wherein the inputs processed through the first processing modules of the processing engines are a subset of features selected from a plurality of training examples in a training set.

12 . The computer-implemented method of claim 11 , wherein the inputs processed through the first processing modules of the processing engines are a subset of the plurality of the training examples in the training set.

13 . The computer-implemented method of claim 1 , further including:

selecting and encoding inputs for a particular first processing module based at least on an architecture of the particular first processing module and/or a task performed by the particular first processing module.

14 . The computer-implemented method of claim 1 , further including:

using parallel processing for performing the training of the plurality of processing engines.

15 . The computer-implemented method of claim 1 , wherein the first processing modules have different architectures and/or different weights.

16 . The computer-implemented method of claim 1 , wherein the second processing modules are copies of each other such that they have a same architecture and/or same weights.

17 . A system for aggregating feature spaces from disparate data silos to execute joint prediction tasks, comprising:

a plurality of prediction engines, respective prediction engines in the plurality of prediction engines having respective encoders and respective decoders;

a plurality of data silos, respective data silos in the plurality of data silos having respective feature spaces that have input features for an overlapping population that spans the respective feature spaces;

a bus system connected to the plurality of prediction engines and configurable to partition the respective prediction engines into respective processing pipelines, and block input feature exchange via the bus system between an encoder within a particular processing pipeline and encoders outside the particular processing pipeline;

a memory access controller connected to the bus system and configurable to confine access of the encoder within the particular processing pipeline to input features of a feature space of a data silo allocated to the particular processing pipeline, and to allow access of a decoder within the particular processing pipeline to encoding generated by the encoder within the particular processing pipeline and to encodings generated by the encoders outside the particular processing pipeline; and

a joint prediction generator connected to the plurality of prediction engines and configurable to process input features from the respective feature spaces of the respective data silos through the respective encoders of corresponding allocated processing pipelines to generate respective encodings, to combine the respective encodings across the allocated processing pipelines to generate combined encodings, and to process the combined encodings through the respective decoders to generate a unified prediction for members of the overlapping population.

18 . A system, comprising:

a joint prediction generator connected to a plurality of prediction engines having respective encoders and respective decoders that are configurable to process input features from respective feature spaces of respective data silos through the respective encoders to generate respective encodings, to combine the respective encodings to generate combined encodings, and to process the combined encodings through the respective decoders to generate a unified prediction for members of an overlapping population that spans the respective feature spaces.

19 . A system for aggregating feature spaces from disparate data silos to execute joint training tasks, comprising:

a plurality of prediction engines, respective prediction engines in the plurality of prediction engines having respective encoders and respective decoders configurable to generate gradients during training;

a plurality of data silos, respective data silos in the plurality of data silos having respective feature spaces that have input features for an overlapping population that spans the respective feature spaces, the input features configurable as training samples for use in the training;

a bus system connected to the plurality of prediction engines and configurable to partition the respective prediction engines into respective processing pipelines, and block training sample exchange and gradient exchange via the bus system during the training between an encoder within a particular processing pipeline and encoders outside the particular processing pipeline;

a memory access controller connected to the bus system and configurable to confine access of the encoder within the particular processing pipeline to input features of a feature space of a data silo allocated as training samples to the particular processing pipeline and to gradients generated from the training of the encoder within the particular processing pipeline, and to allow access of a decoder within the particular processing pipeline to gradients generated from the training of the decoder within the particular processing pipeline and to gradients generated from the training of decoders outside the particular processing pipeline; and

a joint trainer connected to the plurality of prediction engines and configurable to process, during the training, input features from the respective feature spaces of the respective data silos through the respective encoders of corresponding allocated processing pipelines to generate corresponding encodings, to combine the corresponding encodings across the processing pipelines to generate combined encodings, to process the combined encodings through the respective decoders to generate respective predictions for members of the overlapping population, to generate a combined gradient set from respective gradients of the respective decoders generated based on the respective predictions, to generate respective gradients of the respective encoders based on the combined encodings, to update the respective decoders based on the combined gradient set, and to update the respective encoders based on the respective gradients.

20 . A system, comprising:

a joint trainer connected to a plurality of prediction engines have respective encoders and respective decoders that are configurable to process, during training, input features from respective feature spaces of respective data silos through the respective encoders to generate respective encodings, to combine the respective encodings across encoders to generate combined encodings, to process the combined encodings through the respective decoders to generate respective predictions for members of an overlapping population, to generate a combined gradient set from respective gradients of the respective decoders generated based on the respective predictions, to generate respective gradients of the respective encoders based on the combined encodings, to update the respective decoders based on the combined gradient set, and to update the respective encoders based on the respective gradients.