Copula optimization method and apparatus for identifying and detecting threats to an enterprise or e-commerce system and other applications
Methods and apparatuses employing copula optimization in building multivariate statistical models for identifying and detecting threats to an enterprise or e-commerce system are disclosed, including grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; extracting one or more features from the grouped log lines into one or more features tables; using one or more statistical models on the one or more features tables to identify statistical outliers and using the one or more rules on incoming enterprise or e-commerce system data traffic to detect threats to the enterprise or e-commerce system. Other embodiments are described and claimed.
1. A method for identifying and detecting threats to an enterprise or e-commerce system, the method comprising: grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; extracting one or more features from the grouped log lines into one or more features tables; using one or more statistical models on the one or more features tables to identify statistical outliers; and for a Copula statistical model, estimating the marginal probability distribution of a feature using a nonparametric kernel density determination using a Gaussian kernel estimation step, said Gaussian kernel estimation step comprising the step of setting a bandwidth of said Gaussian kernel, and further setting said bandwidth using a Scott's rule of thumb bandwidth setting process.
2. The method of claim 1 , further comprising the step of applying a transformation module to at least a subset of said features for generating non-ordinal variables, in the event that said feature generator produces a plurality of discrete ordinal variables.
3. The method of claim 2 , further comprising the step of adding Gaussian noise to said discrete ordinal variables for producing an essentially continuous valued feature, said essentially continuous valued feature exhibiting an essentially uniform cumulative distribution function.
4. The method of claim 3 , wherein the step of adding said Gaussian noise comprises determining a customized noise value by adding the individual value of a feature and a second value, wherein the second value is based on a variance of a Gaussian distribution for adding said noise, wherein said variance of said Gaussian distribution is determined by dividing a “signal power value” by a desired “signal-to-power ratio” value.
5. The method of claim 4 , further comprising the step of separately estimating said “signal power value” for each feature to yield a customized noise value.
6. A system for identifying and detecting threats to an enterprise or e-commerce system, comprising: a processor memory for storing instructions for identifying and detecting threats to an enterprise or e-commerce system; a computer processor for executing said instructions for identifying and detecting threats to an enterprise or e-commerce system, said instructions comprising: instructions for grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; instructions for extracting one or more features from the grouped log lines into one or more features tables; instructions for using one or more statistical models on the one or more features tables to identify statistical outliers; and instructions for applying a Copula statistical model for estimating the marginal probability distribution of a feature using a nonparametric kernel density determination using a Gaussian kernel estimation step, said Gaussian kernel estimation step comprising the step of setting a bandwidth of said Gaussian kernel, and further setting said bandwidth using a Scott's rule of thumb bandwidth setting process.
7. The system of claim 6 , further comprising instructions for applying a transformation module to at least a subset of said features for generating non-ordinal variables, in the event that said feature generator produces a plurality of discrete ordinal variables.
8. The system of claim 7 , further comprising instructions for adding Gaussian noise to said discrete ordinal variables for producing an essentially continuous valued feature, said essentially continuous valued feature exhibiting an essentially uniform cumulative distribution function.
9. The system of claim 8 , further comprising instructions for adding said Gaussian noise, wherein said instructions comprise determining a customized noise value by adding the individual value of a feature and a second value, wherein the second value is based on a variance of a Gaussian distribution for adding said noise, wherein said variance of said Gaussian distribution is determined by dividing a “signal power” value by a desired “signal-to-power ratio” value.
10. The system of claim 9 , further comprising instructions for separately estimating said “signal power” value for each feature to yield a customized noise value.
11. A networked enterprise or e-commerce system comprising a threat detection and identification system for identifying and detecting threats, to a plurality of computing systems of the networked enterprise or e-commerce system, wherein the plurality of computing systems are networked over a common communications network for communicating with one another in a secure computing environment, wherein the threat detection and identification system comprises: a processor memory for storing instructions for identifying and detecting threats to an enterprise or e-commerce system; a computer processor for executing said instructions for identifying and detecting threats to an enterprise or e-commerce system, said instructions comprising: instructions for grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; instructions for extracting one or more features from the grouped log lines into one or more features tables; instructions for using one or more statistical models on the one or more features tables to identify statistical outliers; and instructions for applying a Copula statistical model for estimating the marginal probability distribution of a feature using a nonparametric kernel density determination using a Gaussian kernel estimation step, said Gaussian kernel estimation step comprising the step of setting a bandwidth of said Gaussian kernel, and further setting said bandwidth using a Scott's rule of thumb bandwidth setting process.
12. The networked enterprise or e-commerce system of claim 11 , wherein said system for identifying and detecting threats further comprises instructions for applying a transformation module to at least a subset of said features for generating non-ordinal variables, in the event that said feature generator produces a plurality of discrete ordinal variables.
13. The networked enterprise or e-commerce system of claim 12 , wherein said system for identifying and detecting threats further comprises further comprises instructions for adding Gaussian noise to said discrete ordinal variables for producing an essentially continuous valued feature, said essentially continuous valued feature exhibiting an essentially uniform cumulative distribution function.
14. The networked enterprise or e-commerce system of claim 13 , wherein said system for identifying and detecting threats further comprises further comprising instructions for adding said Gaussian noise, wherein said instructions comprise determining a customized noise value by adding the individual value of a feature and a second value, wherein the second value is based on a variance of a Gaussian distribution for adding said noise, wherein said variance of said Gaussian distribution is determined by dividing a “signal power value” by a desired “signal-to-power ratio” value.
15. The networked enterprise or e-commerce system of claim 14 , wherein said system for identifying and detecting threats further comprises instructions for separately estimating said “signal power value” for each feature to yield a customized noise value.