Neural networks with subdomain training
Heterogenous neural networks are disclosed that have activation functions that hold multi-variable equations. These variables can be passed from one neuron to another. The neurons may be laid out in a topologically similar fashion to a physical system that the heterogenous neural network is modeling. A neural network may have inputs of more than one type. Only a portion of the inputs (a subdomain) may be optimized In such an instance, the neural network may run forward, backpropagate to all inputs, and then perform optimization only on those inputs which will be optimized.
1 . A system for optimizing a heterogenous neural network, comprising: a processor; a memory in operational communication with the processor, a neural network with at least one neuron input, which resides at least partially in the memory, the neural network comprising neurons with type one inputs originating into the neurons themselves and not transferred along to other neurons, neurons with type two inputs originating from the at least one neuron input and transferred along to other neurons, and a neural network optimizer including instructions residing in memory which are executable by the processor to perform a method which includes:
propagating type two inputs of a second type forward through the neural network;
calculating a cost function based on 1) values within the neurons within the neural network and 2) desired values;
backpropagating a gradient of the cost function through the neural network;
using optimization to reduce error only for the type two inputs;
updating the type two inputs to produce an optimized output of the heterogenous neural network; and
using the optimized output to produce at least one equipment control action;
wherein a second neuron represents a first building portion, wherein a third neuron represents a second building portion, wherein the first building portion and the second building portion are connected, and wherein the second neuron and the third neuron are connected.
2 . The system of claim 1 , wherein the cost function comprises a weighted series of differences between output of the neural network and a time series of zone sensor values.
3 . The system of claim 2 , wherein weights of the weighted series of differences are adjusted based on time distance from a first time value.
4 . The system of claim 1 , wherein type one inputs are temporary value inputs and type two inputs are permanent value inputs.
5 . The system of claim 1 , wherein at least one of the type one inputs and at least one of the type two inputs in at least one neuron are used in an activation function of the at least one neuron.
6 . The system of claim 1 , wherein the heterogenous neural network has at least a first level connecting to a second level, the second level connected to a third level, and wherein at least one signal from at least neuron travels from the first level to the third level.
7 . A method for optimizing a neural network, comprising:
propagating type two inputs through neurons of the neural network, wherein the type two inputs originate from at least one neuron input and are transferred along to other neurons;
using type one inputs to determine activation functions within the neurons, the type one inputs originating into the neurons themselves and not transferred along to other neurons;
calculating a cost function based on 1) values of the neurons within the neural network and 2) desired values;
backpropagating a gradient of the cost function through the inputs to produce an optimized output of the neural network;
updating the type two inputs;
and using the optimized output to produce at least one equipment control action;
wherein a second neuron represents a first building portion, wherein a third neuron represents a second building portion, wherein the first building portion and the second building portion are connected, and wherein the second neuron and the third neuron are connected.
8 . The method of claim 7 , wherein an activation function of at least one neuron comprises a multi-variable equation.
9 . The method of claim 8 , wherein the at least one neuron has an activation function, and wherein the activation function uses a variable from a different neuron.
10 . The method of claim 8 , wherein at least one of the type one inputs and at least one of the type two inputs in at least one neuron are used in an activation function of the at least one neuron.
11 . The method of claim 10 , wherein the neurons have at least one activation function, and wherein the at least one activation function comprises a multi-variable equation with at least one variable of a type one input and at least one variable of a type two input.
12 . They method of claim 10 , wherein the neurons have at least one activation function, and wherein the at least one activation function comprises multiple multi-variable equations.
13 . The method of claim 7 , wherein the neural network has at least a first level connecting to a second level, the second level connected to a third level, and wherein at least one signal from at least neuron travels from the first level to the third level.
14 . A non-transitory computer-readable storage medium configured with data and with instructions that upon execution by at least one processor in a controller computer system having computer hardware, programmable memory, and a heterogenous neural network with at least one neuron input in programable memory, the heterogenous neural network having neurons with type one inputs originating into the neurons themselves and not transferred to other neurons, neurons with type two inputs originating from the at least one neuron input and transferred along to other neurons, and a neural network optimizer including instructions residing in memory which are executable by the at least one processor to perform a technical process for neural network subdomain training, the technical process comprising:
propagating type two inputs through the heterogenous neural network;
calculating a cost function based on 1) values within neurons within the heterogenous neural network and 2) desired values;
backpropagating a gradient of the cost function;
updating the type two inputs to produce an optimized output of the heterogenous neural network;
and using the optimized output to produce at least one equipment control action;
wherein a second neuron represents a first building portion, wherein a third neuron represents a second building portion, wherein the first building portion and the second building portion are connected, and wherein the second neuron and the third neuron are connected.
15 . The non-transitory computer-readable storage medium of claim 14 , wherein an activation function of a neuron comprises a multi-variable equation.
16 . The non-transitory computer-readable storage medium of claim 14 , wherein the heterogenous neural network has at least a first level connecting to a second level, the second level connected to a third level, and wherein at least one signal from at least neuron travels from the first level to the third level.
17 . The non-transitory computer-readable storage medium of claim 14 , wherein at least one of the type one inputs and at least one of the type two inputs in at least one neuron are used in an activation function of the at least one neuron.