Training neural networks using evolution based strategies and novelty search
View Patent ↗Systems and methods are disclosed herein for selecting a parameter vector from a set of parameter vectors for a neural network and generating a plurality of copies of the parameter vector. The systems and methods generate a plurality of modified parameter vectors by perturbing each copy of the parameter vector with a different perturbation seed, and determine, for each respective modified parameter vector, a respective measure of novelty. The systems and methods determine an optimal new parameter vector based on each respective measure of novelty for each respective one of the plurality of modified parameter vectors, and determine behavior characteristics of the new parameter vector. The systems and methods store the behavior characteristics of the new parameter vector in an archive.
1. A computer-implemented method comprising:
initializing a set of parameter vectors for a neural network;
determining behavior characterizations for the set of parameter vectors;
storing the determined behavior characterizations in an archive;
selecting a parameter vector from the set of parameter vectors;
generating a plurality of copies of the selected parameter vector;
generating a plurality of modified parameter vectors by perturbing each copy of the copies of the selected parameter vector with a different perturbation seed of a plurality of perturbation seeds;
determining, for each respective one of the plurality of the modified parameter vectors, a respective measure of novelty by:
selecting k-nearest neighbors of the behavior characterization associated with the respective modified parameter vector; and
computing the average distance between the behavior characterization of the respective modified parameter vector and the selected k-nearest neighbors;
determining, for each respective one of the plurality of the modified parameter vectors, a probability by computing each respective one of the plurality of the modified parameter vector's novelty normalized by the sum of novelty across the plurality of the modified parameter vectors;
selecting, based on the determined probabilities, a modified parameter vector of the plurality of modified parameter vectors as a new selected parameter vector;
performing an update to the new selected parameter vector by computing a gradient of expected novelty with respect to the new selected parameter vector, resulting in an updated parameter vector;
determining behavior characterization of the updated parameter vector; and
storing the behavior characterization of the updated parameter vector in the archive.
2. The computer-implemented method of claim 1 , wherein determining, for each respective one of the plurality of modified parameter vectors, a respective measure of novelty comprises, for a respective modified parameter vector of the plurality of modified parameter vectors:
extracting a parameter of the respective modified parameter vector;
comparing the parameter to each corresponding parameter of each parameter vector of the set of parameter vectors;
determining a degree to which the parameter corresponds to each corresponding parameter; and
determining the respective measure of novelty based on the degree to which the parameter corresponds to each corresponding parameter.
3. The computer-implemented method of claim 2 , wherein the respective measure of novelty has an inverse correlation to the degree to which the parameter corresponds to each corresponding parameter.
4. The computer-implemented method of claim 1 , further comprising:
determining, for each respective one of the plurality of modified parameter vectors, a respective measure of fitness, wherein determining the new selected parameter vector is further based on each respective measure of fitness for each respective one of the plurality of modified parameter vectors.
5. The computer-implemented method of claim 4 , wherein determining, for each respective one of the plurality of modified parameter vectors, a respective measure of fitness comprises, for a respective modified parameter vector of the plurality of modified parameter vectors:
executing the neural network corresponding to the modified parameter vector;
determining a result value based on the output of executing the neural network; and
determining a measure of reward based on a comparison of the result value to a target result value.
6. The computer-implemented method of claim 4 , wherein determining the new selected parameter vector comprises:
applying a first weighting to each respective one of the plurality of modified parameter vectors based on the respective measure of fitness, and applying a second weighting to each respective one of the plurality of modified parameter vectors based on the respective measure of novelty; and
determining the new selected parameter vector based on the applied weightings.
7. The computer-implemented method of claim 1 , further comprising:
receiving sensor data from one or more sensors, wherein the set of parameter vectors is derived from the sensor data; and
performing an action based on output of the neural network, wherein the output comprises the behavior characteristics of the new parameter vector.
8. The computer-implemented method of claim 1 , wherein generating the plurality of modified parameter vectors comprises:
transmitting each respective copy of the plurality of copies of the parameter vector to a different respective worker system of a plurality of worker systems, wherein each respective worker system of the plurality of worker systems generates each modified parameter vector of the plurality of modified parameter vectors.
9. A non-transitory computer-readable storage medium storing executable computer program code that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
initializing a set of parameter vectors for a neural network;
determining behavior characterizations for the set of parameter vectors;
storing the determined behavior characterizations in an archive;
selecting a parameter vector from the set of parameter vectors;
generating a plurality of copies of the selected parameter vector;
generating a plurality of modified parameter vectors by perturbing each copy of the copies of the selected parameter vector with a different perturbation seed of a plurality of perturbation seeds;
determining, for each respective one of the plurality of the modified parameter vectors, a respective measure of novelty by:
selecting k-nearest neighbors of the behavior characterization associated with the respective modified parameter vector; and
computing the average distance between the behavior characterization of the respective modified parameter vector and the selected k-nearest neighbors;
determining, for each respective one of the plurality of the modified parameter vectors, a probability by computing each respective one of the plurality of the modified parameter vector's novelty normalized by the sum of novelty across the plurality of the modified parameter vectors;
selecting, based on the determined probabilities, a modified parameter vector of the plurality of modified parameter vectors a new selected parameter vector;
performing an update to the new selected parameter vector by computing a gradient of expected novelty with respect to the new selected parameter vector, resulting in an updated parameter vector;
determining behavior characterization of the updated parameter vector; and
storing the behavior characterization of the updated parameter vector in the archive.
10. The non-transitory computer-readable storage medium of claim 9 , wherein determining, for each respective one of the plurality of modified parameter vectors, a respective measure of novelty comprises, for a respective modified parameter vector of the plurality of modified parameter vectors:
extracting a parameter of the respective modified parameter vector;
comparing the parameter to each corresponding parameter of each parameter vector of the set of parameter vectors;
determining a degree to which the parameter corresponds to each corresponding parameter; and
determining the respective measure of novelty based on the degree to which the parameter corresponds to each corresponding parameter.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the respective measure of novelty has an inverse correlation to the degree to which the parameter corresponds to each corresponding parameter.
12. The non-transitory computer-readable storage medium of 9 , wherein the program code, when executed by the one or more processors, further causes the one or more processors to perform operations comprising:
determining, for each respective one of the plurality of modified parameter vectors, a respective measure of fitness, wherein determining the new selected parameter vector is further based on each respective measure of fitness for each respective one of the plurality of modified parameter vectors.
13. The non-transitory computer-readable storage medium of claim 12 , wherein determining, for each respective one of the plurality of modified parameter vectors, a respective measure of fitness comprises, for a respective modified parameter vector of the plurality of modified parameter vectors:
executing the neural network corresponding to the modified parameter vector;
determining a result value based on the output of executing the neural network; and
determining a measure of reward based on a comparison of the result value to a target result value.
14. The non-transitory computer-readable storage medium of claim 12 , wherein determining the new selected parameter vector comprises:
applying a first weighting to each respective one of the plurality of modified parameter vectors based on the respective measure of fitness, and applying a second weighting to each respective one of the plurality of modified parameter vectors based on the respective measure of novelty; and
determining the new selected parameter vector based on the applied weightings.
15. The non-transitory computer-readable storage medium of claim 9 , wherein the program code, when executed by the one or more processors, further causes the one or more processors to perform operations comprising:
receiving sensor data from one or more sensors, wherein the set of parameter vectors is derived from the sensor data; and
performing an action based on output of the neural network, wherein the output comprises the behavior characteristics of the new parameter vector.
16. The non-transitory computer-readable storage medium of claim 9 , wherein generating the plurality of modified parameter vectors comprises:
transmitting each respective copy of the plurality of copies of the parameter vector to a different respective worker system of a plurality of worker systems, wherein each respective worker system of the plurality of worker systems generates each modified parameter vector of the plurality of modified parameter vectors.
17. A computer system comprising memory with instructions encoded thereon, and one or more processors that, when executing the instructions, are caused to perform operations comprising:
initializing a set of parameter vectors for a neural network;
determining behavior characterizations for the set of parameter vectors;
storing the determined behavior characterizations in an archive;
selecting a parameter vector from the set of parameter vectors;
generating a plurality of copies of the selected parameter vector;
generating a plurality of modified parameter vectors by perturbing each copy of the copies of the selected parameter vector with a different perturbation seed of a plurality of perturbation seeds;
determining, for each respective one of the plurality of the modified parameter vectors, a respective measure of novelty by:
selecting k-nearest neighbors of the behavior characterization associated with the respective modified parameter vector; and
computing the average distance between the behavior characterization of the respective modified parameter vector and the selected k-nearest neighbors;
determining, for each respective one of the plurality of the modified parameter vectors, a probability by computing each respective one of the plurality of the modified parameter vector's novelty normalized by the sum of novelty across the plurality of the modified parameter vectors;
selecting, based on the determined probabilities, a modified parameter vector of the plurality of modified parameter vectors a new selected parameter vector;
performing an update to the new selected parameter vector by computing a gradient of expected novelty with respect to the new selected parameter vector, resulting in an updated parameter vector;
determining behavior characterization of the updated parameter vector; and
storing the behavior characterization of the updated parameter vector in the archive.
18. The computer system of claim 17 , wherein determining, for each respective one of the plurality of modified parameter vectors, a respective measure of novelty comprises, for a respective modified parameter vector of the plurality of modified parameter vectors:
extracting a parameter of the respective modified parameter vector;
comparing the parameter to each corresponding parameter of each parameter vector of the set of parameter vectors;
determining a degree to which the parameter corresponds to each corresponding parameter; and
determining the respective measure of novelty based on the degree to which the parameter corresponds to each corresponding parameter.