Mining non-linear dependencies via a neighborhood mixture model
A computer-implemented method for simultaneous metric learning and variable selection in non-linear regression is presented. The computer-implemented method includes introducing a dataset and a target variable, creating a univariate neighborhood probability map for each reference point of the dataset, and determining a pairwise distance between each reference point and other points within the dataset. The computer-implemented method further includes computing a Hessian matrix of a quadratic programming (QP) problem, performing optimization of the QP problem, re-weighing data derived from the optimization of the QP problem, and performing non-linear regression on the re-weighed data.
1. A computer-implemented method executed on a processor of a computer for simultaneous metric learning and variable selection in non-linear regression, the method comprising:
introducing a dataset having a plurality of input variables and a target variable;
learning a non-negative re-scaling factor from the plurality of input variables;
building a univariate neighborhood probability map for each reference point of the dataset by defining a neighborhood probability matrix for a neighborhood mixture model;
determining a pairwise distance between each reference point and other points within the dataset;
based on determined pairwise distances, computing a Hessian matrix of a quadratic programming (QP) problem;
performing optimization of the QP problem, the optimization being a concave-convex optimization;
re-weighing features derived from the optimization of the QP problem; and
performing non-linear regression on the re-weighed data to generate a structural map laying out a structural organization of identified non-linear relationships between the plurality of input variables and the target variable by employing the neighborhood mixture model which approximates multivariate close neighbors with a mixture of neighborhood probability across each individual dimension,
wherein the identified non-linear relationships are extracted only from a subset of relevant input variables of the plurality of input variables to reduce a number of computations of the QP problem to improve functionality of the computer running the simultaneous metric learning and variable selection and to increase accuracy of the generated structural map.
2. The method of claim 1 , wherein the neighborhood probability matrix is given by:
P
jd
(
i
)
=
1
2
h
id
exp
(
-
(
x
i
[
d
]
-
x
j
[
d
]
)
2
h
id
)
where h id is a width of a Gaussian function for the ith reference point and d th dimension, and x i , x j are reference samples in the d th dimension.
3. The method of claim 1 , wherein the pairwise distance is given by:
| x i −x j | ⊙ q =[| x i[1] −x j[1] | q ,|x i[2] −x j[2] | q , . . . ,|x i[d] −x j[d] | q ] T
where x i , x j are reference samples, T denotes transpose to a row vector, and q denotes a dimensional vector of non-negative weights.
4. The method of claim 1 , wherein local geometric constraints are selectively enforced on an optimized neighborhood.
5. The method of claim 1 , wherein the non-linear relationships between the plurality of input variables and the target variable are identified by using generic criteria that measure learnability from an input variable of the plurality of input variables to an output variable.
6. The method of claim 1 , wherein re-weighing the data allows for learning an optimal non-negative scaling for each variable in the dataset.
7. The method of claim 1 , wherein a neighborhood structure is optimized simultaneously with a metric parameter.
8. A system for simultaneous metric learning and variable selection in non-linear regression, the system comprising:
a memory; and
a processor in communication with the memory, wherein the processor is configured to:
introduce a dataset having a plurality of input variables and a target variable;
learn a non-negative re-scaling factor from the plurality of input variables;
build a univariate neighborhood probability map for each reference point of the dataset by defining a neighborhood probability matrix for a neighborhood mixture model;
determine a pairwise distance between each reference point and other points within the dataset;
based on determined pairwise distances, compute a Hessian matrix of a quadratic programming (QP) problem;
perform optimization of the QP problem, the optimization being a concave-convex optimization;
re-weigh features derived from the optimization of the QP problem; and
perform non-linear regression on the re-weighed data to generate a structural map laying out a structural organization of identified non-linear relationships between the plurality of input variables and the target variable by employing the neighborhood mixture model which approximates multivariate close neighbors with a mixture of neighborhood probability across each individual dimension,
wherein the identified non-linear relationships are extracted only from a subset of relevant input variables of the plurality of input variables to reduce a number of computations of the QP problem to improve functionality of a computer running the simultaneous metric learning and variable selection and to increase accuracy of the generated structural map.
9. The system of claim 8 , wherein the neighborhood probability matrix is given by:
P
jd
(
i
)
=
1
2
h
id
exp
(
-
(
x
i
[
d
]
-
x
j
[
d
]
)
2
h
id
)
where h id is a width of a Gaussian function for the ith reference point and d th dimension, and x i , x j are reference samples in the d th dimension.
10. The system of claim 8 , the pairwise distance is given by:
| x i −x j | ⊙ q =[| x i[1] −x j[1] | q ,|x i[2] −x j[2] | q , . . . ,|x i[d] −x j[d] | q ] T
where x i , x j are reference samples, T denotes transpose to a row vector, and q denotes a dimensional vector of non-negative weights.
11. The system of claim 8 , wherein local geometric constraints are selectively enforced on an optimized neighborhood.
12. The system of claim 8 , wherein the non-linear relationships between the plurality of input variables and the target variable are identified by using generic criteria that measure learnability from an input variable of the plurality of input variables to an output variable.
13. The system of claim 8 , wherein re-weighing the data allows for learning an optimal non-negative scaling for each variable in the dataset.
14. The system of claim 8 , wherein a neighborhood structure is optimized simultaneously with a metric parameter.
15. A non-transitory computer-readable storage medium comprising a computer-readable program for simultaneous metric learning and variable selection in non-linear regression, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
introducing a dataset having a plurality of input variables and a target variable;
learning a non-negative re-scaling factor from the plurality of input variables;
building a univariate neighborhood probability map for each reference point of the dataset by defining a neighborhood probability matrix for a neighborhood mixture model;
determining a pairwise distance between each reference point and other points within the dataset;
based on determined pairwise distances, computing a Hessian matrix of a quadratic programming (QP) problem;
performing optimization of the QP problem, the optimization being a concave-convex optimization;
re-weighing features derived from the optimization of the QP problem; and
performing non-linear regression on the re-weighed data to generate a structural map laying out a structural organization of identified non-linear relationships between the plurality of input variables and the target variable by employing the neighborhood mixture model which approximates multivariate close neighbors with a mixture of neighborhood probability across each individual dimension,
wherein the identified non-linear relationships are extracted only from a subset of relevant input variables of the plurality of input variables to reduce a number of computations of the QP problem to improve functionality of the computer running the simultaneous metric learning and variable selection and to increase accuracy of the generated structural map.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the neighborhood probability matrix is given by:
P
jd
(
i
)
=
1
2
h
id
exp
(
-
(
x
i
[
d
]
-
x
j
[
d
]
)
2
h
id
)
where h id is a width of a Gaussian function for the ith reference point and d th dimension, and x i , x j are reference samples in the d th dimension.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the pairwise distance is given by:
| x i −x j | ⊙ q =[| x i[1] −x j[1] | q ,|x i[2] −x j[2] | q , . . . ,|x i[d] −x j[d] | q ] T
where x i , x j are reference samples, T denotes transpose to a row vector, and q denotes a dimensional vector of non-negative weights.
18. The non-transitory computer-readable storage medium of claim 15 , wherein local geometric constraints are selectively enforced on an optimized neighborhood.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the non-linear relationships between the plurality of input variables and the target variable are identified by using generic criteria that measure learnability from an input variable of the plurality of input variables to an output variable.
20. The non-transitory computer-readable storage medium of claim 15 , wherein a neighborhood structure is optimized simultaneously with a metric parameter.