Architectural augmentation of neural networks using evaluated specialty node units
A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.
1 . A computer-implemented method comprising:
iteratively training, at least partially, by a programmed computer system that comprises one or more processor cores, via machine learning, a neural network to perform a training task; and
after the training:
generating, by the programmed computer system, a plurality of candidate specialty node units for the neural network, each comprising one or more nodes, and wherein each candidate specialty node unit has a specialty type selected from the group consisting of: a sparse node unit, a template node unit, a local autoencoder unit, a discriminator unit, a node unit with memory, a program node unit, and a second-order function node unit;
evaluating, by the programmed computer system, performance of the plurality of candidate specialty node units on the training task using a subset of training data;
selecting, by the programmed computer system, one or more of the candidate specialty node units based on the evaluation;
inserting, by the programmed computer system, the selected one or more candidate specialty node units into the neural network to create a modified neural network, wherein each of the selected on or more candidate specialty node units, when inserted, is functionally integrated into the neural network at an insertion location determined based on task-specific criteria; and
resuming, by the programmed computer system, iterative training of the modified neural network using training data.
2 . The method of claim 1 , wherein evaluating performance of the plurality of candidate specialty node units comprises computing a change in loss of the neural network when each candidate specialty node unit is provisionally inserted into the neural network.
3 . The method of claim 1 , wherein evaluating performance of the plurality of candidate specialty node units comprises computing a change in accuracy of the neural network when each candidate specialty node unit is provisionally inserted into the neural network.
4 . The method of claim 1 , wherein selecting one or more of the candidate specialty node units comprises excluding a candidate specialty node unit whose insertion results in degradation of neural network performance beyond a threshold value.
5 . The method of claim 1 , wherein the insertion location for a selected specialty node unit is determined based on gradient magnitude at nodes in the neural network.
6 . The method of claim 1 , wherein the insertion location for a selected specialty node unit is determined based on identification of a bottleneck in the neural network.
7 . The method of claim 1 , wherein the insertion location for a selected specialty node unit is determined based on activation sparsity of nodes in the neural network.
8 . The method of claim 1 , further comprising training each specialty node unit using a subset of the training data selected to emphasize a subdomain of the training task.
9 . The method of claim 8 , further comprising assigning a specialization tag to each selected specialty node unit based on the subdomain emphasized during training of that specialty node unit.
10 . The method of claim 1 , wherein a program node unit comprises a symbolic computational graph external to the neural network.
11 . The method of claim 1 , wherein a second-order function node unit is configured to compute a transformation of a parameter of another node within the neural network.
12 . The method of claim 1 , further comprising:
identifying a region within the neural network exhibiting sparse activation or a gradient bottleneck; and
training a specialty node unit using training data corresponding to a subdomain of the training task aligned with that region,
wherein the trained specialty node unit is inserted into the identified region of the neural network.
13 . The method of claim 1 , further comprising freezing weights of at least one inserted specialty node unit during resumed training of the modified neural network.
14 . The method of claim 1 , wherein generating the plurality of candidate specialty node units comprises instantiating the candidate specialty node units from a predefined library of reusable module templates.
15 . The method of claim 1 , further comprising logging insertion decisions and corresponding task performance results to a module registry configured to support reuse of specialty node units in other neural network training workflows.
16 . A computer system comprising:
one or more processor cores; and
a non-transitory computer-readable medium storing instructions that, when executed by the one or more processor cores, cause the computer system to perform operations comprising:
iteratively training, at least partially, via machine learning, a neural network to perform a training task;
after the training:
generating a plurality of candidate specialty node units for the neural network, each comprising one or more nodes, and wherein each candidate specialty node unit has a specialty type selected from the group consisting of: a sparse node unit, a template node unit, a local autoencoder unit, a discriminator unit, a node unit with memory, a program node unit, and a second-order function node unit;
evaluating performance of the plurality of candidate specialty node units on the training task using a subset of training data;
selecting one or more of the candidate specialty node units based on the evaluation;
inserting the selected one or more candidate specialty node units into the neural network to create a modified neural network, wherein each of the selected on or more candidate specialty node units, when inserted, is functionally integrated into the neural network at an insertion location determined based on task-specific criteria; and
resuming iterative training of the modified neural network using training data.
17 . The computer system of claim 16 , wherein the non-transitory computer-readable medium stores instructions that, when executed by the one or more processor cores, cause the computer system to evaluate performance of the plurality of candidate specialty node units by computing a change in loss of the neural network when each candidate specialty node unit is provisionally inserted into the neural network.
18 . The computer system of claim 16 , wherein the non-transitory computer-readable medium stores instructions that, when executed by the one or more processor cores, cause the computer system to evaluate performance of the plurality of candidate specialty node units by computing a change in accuracy of the neural network when each candidate specialty node unit is provisionally inserted into the neural network.
19 . The computer system of claim 16 , wherein the non-transitory computer-readable medium stores instructions that, when executed by the one or more processor cores, cause the computer system to select one or more of the candidate specialty node units by excluding a candidate specialty node unit whose insertion results in degradation of neural network performance beyond a threshold value.
20 . The computer system of claim 16 , wherein the insertion location for a selected specialty node unit is determined based on gradient magnitude at nodes in the neural network.
21 . The computer system of claim 16 , wherein the insertion location for a selected specialty node unit is determined based on identification of a bottleneck in the neural network.
22 . The computer system of claim 16 , wherein the insertion location for a selected specialty node unit is determined based on activation sparsity of nodes in the neural network.
23 . The computer system of claim 16 , wherein the instructions further cause the computer system to train each specialty node unit using a subset of the training data selected to emphasize a subdomain of the training task.
24 . The computer system of claim 23 , wherein the instructions further cause the computer system to assign a specialization tag to each selected specialty node unit based on the subdomain emphasized during training of that specialty node unit.
25 . The computer system of claim 16 , wherein a program node unit comprises a symbolic computational graph external to the neural network.
26 . The computer system of claim 16 , wherein a second-order function node unit is configured to compute a transformation of a parameter of another node within the neural network.
27 . The computer system of claim 16 , wherein the instructions further cause the computer system to:
identify a region within the neural network exhibiting sparse activation or a gradient bottleneck; and
train a specialty node unit using training data corresponding to a subdomain of the training task aligned with that region,
wherein the trained specialty node unit is inserted into the identified region of the neural network.
28 . The computer system of claim 16 , wherein the instructions further cause the computer system to freeze weights of at least one inserted specialty node unit during resumed training of the modified neural network.
29 . The computer system of claim 16 , wherein the non-transitory computer-readable medium stores instructions that, when executed by the one or more processor cores, cause the computer system to generate the plurality of candidate specialty node units by instantiating the candidate specialty node units from a predefined library of reusable module templates.
30 . The computer system of claim 16 , wherein the instructions further cause the computer system to log insertion decisions and corresponding task performance results to a module registry configured to support reuse of specialty node units in other neural network training workflows.