System and method for implementing modular universal reparameterization for deep multi-task learning across diverse domains
View Patent ↗A process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems is described. The process is based on decomposing the general multi-task learning problem into several fine-grained and equally-sized subproblems, or pseudo-tasks. Training a set of (architecture, task) pairs then corresponds to solving a set of related pseudo-tasks, whose relationships can be exploited by shared functional modules. An efficient search algorithm is introduced for optimizing the mapping between pseudo-tasks and the modules that solve them, while simultaneously training the modules themselves.
1. A machine-learning process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems, comprising:
decomposing, by one or more specially programmed processors, each of the multiple (architecture, task) pairs into equally sized pseudo-tasks;
aligning, by the one or more specially programmed processors pseudo-tasks across the multiple diverse architectures, wherein aligning pseudo-tasks comprises optimizing a mapping between the pseudo-tasks and the functional modules of the diverse architectures that solve the pseudo-tasks; and
sharing, by the one or more specially programmed processors, learned parameters across the aligned pseudo-tasks, wherein each diverse architecture is preserved in performance of its paired task.
2. The process according to claim 1 , wherein the diverse architectures include core layers selected from the following group consisting of 2D convolutional, LSTM, 1D convolutional and Dense.
3. The process according to claim 1 , wherein the optimizing uses a stochastic algorithm.
4. The process according to claim 1 , wherein the sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks is implemented using factorization.
5. The process according to claim 1 , wherein the multiple diverse problems are selected from the group consisting of a vision problem, a sorting problem, a natural language processing problem, a speech problem, a biological problem, a geological problem and an astronomical problem.
6. A machine-learning process for training and sharing functional modules across diverse architectures for performing diverse tasks without changing functional forms of underlying predictive models, comprising:
decomposing, by one or more specially programmed processors, each parameter set for each predictive model into parameter blocks, wherein a parameter block is parameterized by a module;
aligning, by the one or more specially programmed processors, pseudo-tasks in accordance with associated parameter sets across the multiple diverse architectures, wherein the aligning includes optimizing a mapping between the pseudo-tasks and the functional modules that solve the pseudo-tasks; and
sharing, by the one or more specially programmed processors, modules across the diverse architectures in accordance with a predetermined alignment, wherein the diverse architectures perform diverse tasks and the sharing of modules improves performance in each diverse task.
7. The process according to claim 6 , wherein the diverse architectures include core layers selected from the following group consisting of 2D convolutional, LSTM, 1D convolutional and Dense.
8. The process according to claim 6 , wherein the optimizing uses a stochastic algorithm.
9. The process according to claim 6 , wherein the sharing by the one or more specially programmed processors includes sharing learned parameters across the aligned pseudo-tasks.
10. The process according to claim 9 , wherein sharing learned parameters across the aligned pseudo-tasks is implemented using factorization.
11. The process according to claim 6 , wherein the multiple diverse tasks are selected from the group consisting of a vision-related task, a sorting-related task, a natural language processing-related task, a speech-related task, a biological-related task, a geological-related task and an astronomical-related task.
12. A computer-implemented machine learning process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems, comprising:
means for decomposing, by one or more specially programmed processors, each of the multiple (architecture, task) pairs into equally sized pseudo-tasks;
means for aligning, by the one or more specially programmed processors, pseudo-tasks across the multiple diverse architectures, wherein aligning pseudo-tasks comprises optimizing a mapping between the pseudo-tasks and the functional modules of the diverse architectures that solve the pseudo-tasks; and
means for sharing, by the one or more specially programmed processors, learned parameters across the aligned pseudo-tasks, wherein each diverse architecture is preserved in performance of its paired task.
13. The process according to claim 12 , wherein the means of optimizing is a stochastic algorithm.
14. The process according to claim 12 , wherein the means for sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks includes factorization.