Configuration process framework for machine learning models
View Patent ↗Various embodiments are directed to configuring or training deep neural network (DNN) machine learning models comprising one or more hidden layers and an output layer. Various embodiments provide technical advantages in training DNN machine learning models, including improved computational efficiency and guaranteed optimality. In one embodiment, an example method includes identifying a nonlinear-model-based representation for each hidden layer, which may be a Bank of Wiener Models, a nonlinear units of the hidden layer, and/or the like. The method further includes individually and sequentially configuring the hidden layers, each configured by determining a correlation measure (e.g., a correlation ratio) between the layer output and a target signal. Parameters of the particular hidden layer are modified by maximizing the correlation measure to yield maximal correlation over the space of functions. The method further includes performing automated tasks using the DNN machine learning model after configuring its parameters on a training set.
1 . A computer-implemented method for configuring a deep neural network (DNN) machine learning model comprising one or more hidden layers and an output layer, the method comprising:
receiving, using one or more processors, one or more input signals and one or more target signals, wherein each of the one or more target signals corresponds to an input signal;
selecting, using the one or more processors and based at least in part on the one or input signals and one or more automated tasks, a nonlinear-model-based representation for each hidden layer of the DNN machine learning model, wherein the nonlinear-model-based representation comprises one or more parameters configured to be modified to configure a corresponding hidden layer;
sequentially configuring, using the one or more processors, at least a selected subset of the one or more hidden layers of the DNN machine learning model, wherein the selected subset comprises a particular hidden layer, and wherein the particular hidden layer is configured independently from, and before configuring subsequent hidden layers of the selected subset, by:
determining a correlation measure between (i) an output of the particular hidden layer in response to a given input signal, and (ii) a given target signal corresponding to the given input signal,
modifying the one or more parameters of the nonlinear-model-based representation for the particular hidden layer based at least in part on maximizing the correlation measure, and
fixing the one or more modified parameters of the nonlinear-model-based representation for the particular hidden layer such that the one or more modified parameters cannot be modified by configuring a subsequent hidden layer; and
initiating performance of the one or more automated tasks using the DNN machine learning model.
2 . The method of claim 1 , wherein sequentially configuring the one or more hidden layers comprises fixing the modified parameters of the nonlinear-model-based representation for the particular hidden layer before modifying parameters of a nonlinear-model-based representation for a subsequent hidden layer.
3 . The method of claim 1 , wherein the output layer is a final projection layer that is configured subsequent to the sequential configuration of at least the selected subset of the one or more hidden layers of the DNN machine learning model, the output layer being configured based at least in part on a least square projection.
4 . The method of claim 1 , wherein the output layer is a final projection layer that is configured subsequent to the sequential configuration of at least the selected subset of the one or more hidden layers of the DNN machine learning model, the output layer being configured based at least in part on maximizing the correlation measure with the target signal.
5 . The method of claim 1 , wherein the correlation measure is a correlation ratio between (i) the layer output of the particular hidden layer in response to the given input signal, and (ii) the given target signal corresponding to the given input signal.
6 . The method of claim 1 , wherein a nonlinear-model-based representation for a hidden layer comprises a plurality of block-oriented nonlinear models.
7 . The method of claim 6 , wherein at least one of the plurality of block-oriented nonlinear models is a Hammerstein-Wiener model.
8 . The method of claim 1 , wherein at least one hidden layer of the DNN machine learning model is substituted by a nonlinear mapping of the one or more input signals to a reproducing kernel Hilbert space (RKHS) where a linear weighting of a plurality of projections is configured by maximizing the correlation measure with the target signal.
9 . The method of claim 1 , wherein the final projection layer is further configured using a combination of one or more outputs from the one or more hidden layers.
10 . The method of claim 1 , wherein the layer output of the particular hidden layer in response to the given input signal is determined directly from the output of a preceding hidden layer.
11 . The method of claim 1 , wherein the layer output of the particular hidden layer in response to the given input signal is determined based at least in part on a combination of one or more outputs of one or more preceding layers.
12 . An apparatus for configuring a deep neural network (DNN) machine learning model comprising one or more hidden layers and an output layer, and the apparatus comprising at least one processor and at least one memory including program code, the at least one memory and the program code configured to, with the processor, cause the apparatus to at least:
receive one or more input signals and one or more target signals, wherein each of the one or more target signals corresponds to an input signal;
select, based at least in part on the one or input signals and one or more automated tasks, a nonlinear-model-based representation for each hidden layer of the DNN machine learning model, wherein the nonlinear-model-based representation comprises one or more parameters configured to be modified to configure a corresponding hidden layer;
sequentially configure at least a selected subset of the one or more hidden layers of the DNN machine learning model, wherein the selected subset comprises a particular hidden layer, and wherein the particular hidden layer is configured independently from, and before configuring subsequent hidden layers of the selected subset, by:
determining a correlation measure between (i) an output of the particular hidden layer in response to a given input signal, and (ii) a given target signal corresponding to the given input signal,
modifying the one or more parameters of the nonlinear-model-based representation for the particular hidden layer based at least in part on maximizing the correlation measure, and
fixing the one or more modified parameters of the nonlinear-model-based representation for the particular hidden layer such that the one or more modified parameters cannot be modified by configuring a subsequent hidden layer; and
initiate performance of the one or more automated tasks using the DNN machine learning model.
13 . The apparatus of claim 12 , wherein the apparatus sequentially configures the one or more hidden layers by at least fixing the modified parameters of the nonlinear-model-based representation for the particular hidden layer before modifying parameters of a nonlinear-model-based representation for a subsequent hidden layer.
14 . The apparatus of claim 12 , wherein the output layer is a final projection layer that is configured subsequent to the sequential configuration of at least the selected subset of the one or more hidden layers of the DNN machine learning model, the output layer being configured based at least in part on a least square projection.
15 . The apparatus of claim 12 , wherein the output layer is a final projection layer that is configured subsequent to the sequential configuration of at least the selected subset of the one or more hidden layers of the DNN machine learning model, the output layer being configured based at least in part on maximizing the correlation measure with the target signal.
16 . The apparatus of claim 12 , wherein the correlation measure is a correlation ratio between (i) the layer output of the particular hidden layer in response to the given input signal, and (ii) the given target signal corresponding to the given input signal.
17 . The apparatus of claim 12 , wherein a nonlinear-model-based representation for a hidden layer comprises a plurality of block-oriented nonlinear models.
18 . The apparatus of claim 17 , wherein at least one of the plurality of block-oriented nonlinear models is a Hammerstein-Wiener model.
19 . The apparatus of claim 12 , wherein at least one hidden layer of the DNN machine learning model is substituted by a nonlinear mapping of the one or more input signals to a reproducing kernel Hilbert space (RKHS) where a linear weighting of a plurality of projections is configured by maximizing the correlation measure with the target signal.
20 . The apparatus of claim 12 , wherein the final projection layer is further configured using a combination of one or more outputs from the one or more hidden layers.
21 . The apparatus of claim 12 , wherein the layer output of the particular hidden layer in response to the given input signal is determined directly from the output of a preceding hidden layer.
22 . The apparatus of claim 12 , wherein the layer output of the particular hidden layer in response to the given input signal is determined based at least in part on a combination of one or more outputs of one or more preceding layers.
23 . A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:
an executable portion configured to receive one or more input signals and one or more target signals, wherein each of the one or more target signals corresponds to an input signal;
an executable portion configured to select, based at least in part on the one or input signals and one or more automated tasks, a nonlinear-model-based representation for each hidden layer of the DNN machine learning model, wherein the nonlinear-model-based representation comprises one or more parameters configured to be modified to configure a corresponding hidden layer;
an executable portion configured to sequentially configure at least a selected subset of the one or more hidden layers of the DNN machine learning model, wherein the selected subset comprises a particular hidden layer, and wherein the particular hidden layer is configured independently from, and before configuring subsequent hidden layers of the selected subset, by:
determining a correlation measure between (i) an output of the particular hidden layer in response to a given input signal, and (ii) a given target signal corresponding to the given input signal,
modifying the one or more parameters of the nonlinear-model-based representation for the particular hidden layer based at least in part on maximizing the correlation measure, and
fixing the one or more modified parameters of the nonlinear-model-based representation for the particular hidden layer such that the one or more modified parameters cannot be modified by configuring a subsequent hidden layer; and
an executable portion configured to initiate performance of the one or more automated tasks using the DNN machine learning model.