Particle flow training of Bayesian neural network
Discussed herein are devices, systems, and methods for training and operating a Bayesian neural network (BNN). A method can include initializing particles that each individually represent pointwise values of respective NN parameters of NNs that collectively represent a distribution of parameters of the BNN, optimizing, using training particle flow, the particles resulting in optimized distributions for the parameters, determining a prediction distribution using the optimized distributions for the parameters and predictions from each of the NNs, and providing a marginalized distribution representative of the prediction distribution.
1 . A method for training and operating a Bayesian neural network (BNN), the method comprising:
initializing particles that each individually represent pointwise values of sets of respective NN parameters of respective NNs that collectively represent a distribution of parameters of the BNN;
optimizing, using training particle flow that equates internal states and measurements of particle flow with network parameters and truth values, respectively and replaces likelihood of each measurement given an internal state with a likelihood of a value given a prediction, the particles resulting in optimized distributions for the parameters including, for each data, looping through values of a log-homotopy parameter to evolve a log of the joint posterior probability of the data;
determining a prediction distribution using the optimized distributions for the parameters and predictions from each of the NNs; and
providing a marginalized distribution representative of the prediction distribution.
2 . The method of claim 1 , wherein training particle flow includes a particle flow technique used in a Daum-Huang particle filter with internal states and measurements replaced with weights of the NN parameters and truth values, respectively.
3 . The method of claim 1 , wherein training particle flow includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.
4 . The method of claim 3 , wherein training particle flow includes using a diagonal approximation to the Gauss-Newton approximation.
5 . The method of claim 4 , wherein the diagonal approximation includes performing Gauss-Newton approximation per weight.
6 . A non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform operations for training and operating a Bayesian neural network (BNN) comprising:
initializing particles that each individually represent pointwise values of sets of respective NN parameters of respective NNs that collectively represent a distribution of parameters of the BNN;
optimizing, using training particle flow that equates internal states and measurements of particle flow with network parameters and truth values, respectively and replaces likelihood of each measurement given an internal state with a likelihood of a value given a prediction, the particles resulting in optimized distributions for the parameters including, for each data, looping through values of a log-homotopy parameter to evolve a log of the joint posterior probability of the data;
determining a prediction distribution using the optimized distributions for the parameters and predictions from each of the NNs; and
providing a marginalized distribution representative of the prediction distribution.
7 . The non-transitory machine-readable medium of claim 6 , wherein training particle flow includes a particle flow technique used in a Daum-Huang particle filter with internal states and measurements replaced with weights of the NN parameters and the truth values, respectively.
8 . The non-transitory machine-readable medium of claim 6 , wherein training particle flow includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.
9 . The non-transitory machine-readable medium of claim 8 , wherein training particle flow includes using a diagonal approximation to the Gauss-Newton approximation.
10 . The non-transitory machine-readable medium of claim 9 , wherein the diagonal approximation includes performing Gauss-Newton approximation per weight.
11 . A device comprising:
a memory device including instructions stored thereon;
processing circuitry coupled to the memory device, the processing circuitry configured to execute the instructions, the instructions, when executed by the processing circuitry cause the processing circuitry to perform operations for training and operating a Bayesian neural network (BNN), the operations comprising:
initializing particles that each individually represent pointwise values of sets of respective NN parameters of respective NNs that collectively represent a distribution of parameters of the BNN;
optimizing, using training particle flow that equates internal states and measurements of particle flow with network parameters and truth values, respectively and replaces likelihood of each measurement given an internal state with a likelihood of a value given a prediction, the particles resulting in optimized distributions for the parameters including, for each data, looping through values of a log-homotopy parameter to evolve a log of the joint posterior probability of the data;
determining a prediction distribution using the optimized distributions for the parameters and predictions from each of the NNs; and
providing a marginalized distribution representative of the prediction distribution.
12 . The device of claim 11 , wherein training particle flow includes a particle flow technique used in a Daum-Huang particle filter with internal states and measurements replaced with weights of the NN parameters and truth values, respectively.
13 . The device of claim 11 , wherein training particle flow includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.
14 . The device of claim 13 , wherein training particle flow includes using a diagonal approximation to the Gauss-Newton approximation, the diagonal approximation includes performing Gauss-Newton approximation per weight.