Anomaly detection with machine learning

1. A method of detecting an anomaly event that has not frequently been observed in an ongoing event stream of security-related events of one or more organizations, the method including:

evaluating a plurality of production events with production space IDs, including for a production event:

transforming features of the production event into categorical bins of a hash-space;

applying a hash function to the production space ID and the features of the production event as transformed to retrieve likelihood coefficients for the transformed features of the production event and a standard candle for the production space ID;

calculating an anomaly score;

when the anomaly score represents a detected anomaly event, accessing history associated with the production space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior events for the production space ID; and

invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected;

wherein the likelihood coefficients had been calculated by space ID and a standard candle and mapped into the hash-space using a loosely supervised machine learning of observed features in security-related events using a loss function analyzer and recording the standard candle.

2. The method of claim 1 , wherein the loosely supervised machine learning of observed features in security-related events using a loss function analyzer and recording a standard candle, further includes:

transforming training events by assigning the observed features into categorical bins and coding the assigned observed features with a Boolean value as present in their respective categorical bins; and

analyzing the transformed training events using the loss function analyzer, treating the security-related events as having occurred with certainty, requiring the loss function analyzer to analyze the security-related events by a space identifier (ID), and requiring the loss function analyzer to fit the observed features as transformed essentially through an origin.

3. The method of claim 1 , further including generating for display the anomaly event in naturally processed language.

4. The method of claim 1 , further including:

storing a set of coded feature-value pairs and corresponding likelihood coefficients in a persistent storage using a dense feature representation; and

storing the set of coded feature-value pairs and corresponding likelihood coefficients on memory using a sparse feature representation.

5. The method of claim 1 , wherein the loss function analyzer is a stochastic gradient descent (SGD) analyzer.

6. The method of claim 1 , wherein events are annotated with prevalencist probability values of between 0 to 1, indicative of an occurrence frequency of the events.

7. The method of claim 6 , wherein a prevalencist probability value of 0 indicates previously unseen events.

8. The method of claim 6 , wherein a prevalencist probability value of 1 indicates frequently appearing events.

9. The method of claim 6 , further including:

storing the likelihood coefficients and the prevalencist probability values for multiple space IDs of an organization in a hash-space as a tenant activity model, indicative of activity habits of users in the organization; and

updating the tenant activity model with new events to incorporate changes to the activity habits.

10. The method of claim 6 , further including:

storing the likelihood coefficients and the prevalencist probability values for a particular space ID in a hash-space as a user activity model, indicative of activity habits of a user; and

updating the user activity model with new events to incorporate changes to the activity habits.

11. The method of claim 6 , further including:

determining a relative-error ratio for a particular production event with a production space ID based on a predicted prevalencist probability value of the production event and an observed prevalencist probability value of the production event;

determining a standard candle value for the production space ID based on a maximum likelihood coefficient feature-value pair in the production event;

evaluating likelihood coefficients of individual feature-value pairs in the production event and determining one or more lowest likelihood coefficient feature-value pairs in the production event;

calculating an overall likelihood coefficient for the production event based on the one or more lowest likelihood coefficient feature-value pairs; and

determining the production event to be an anomaly event when the relative-error ratio, the standard candle value and the overall likelihood coefficient exceed a threshold.

12. The method of claim 11 , further including distinguishing between a seasoned user and an unseasoned user by:

requiring initialization and analysis of a space ID by the loss function analyzer with the standard candle value; and

maturing the standard candle value of the production space ID to a target value responsive to a threshold number of events received for the space ID, wherein seasoned space IDs have non-zero standard candle values.

13. The method of claim 11 , further including:

clustering a plurality of production events with lowest likelihood coefficient feature-value pairs based on a feature-dimension type; and

generating for display clustered productions events for different feature-dimension types.

14. The method of claim 1 , further including updating tenant and user activity models over time, including maturing and storing frequently occurring anomalous events as normal user activity.

15. A method of detecting anomalies in an event stream of security-related events of one or more organizations, the method including:

receiving a security-related event that includes a space identifier (ID) and features with certain values;

transforming feature-value pairs into categorical bins of a hash-space;

applying a hash function to a combination of the space ID and the transformed feature-value pairs to retrieve likelihood coefficients for the transformed feature-value pairs and a standard candle for the space ID;

scoring the likelihood coefficients and any transformed feature-value pairs that do not have likelihood coefficients because the transformed feature-value pairs that do not have likelihood coefficients were not previously observed for the space ID, in combination with evaluating the standard candle, to produce an anomaly score;

when the anomaly score represents a detected anomaly event, accessing history associated with the space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior security-related events for the space ID; and

invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected.

16. The method of claim 15 , wherein a compressed set of stored likelihood coefficients and stored standard candles for a plurality of users has been expanded into a hash-space.

17. A non-transitory computer readable storage medium impressed with computer program instructions to detect an anomaly event that has not frequently been observed in an ongoing event stream of security-related events of one or more organizations, which computer program instructions, when executed on a processor, implement a method comprising:

evaluating a plurality of production events with production space IDs, including for a production event:

transforming features of the production event into categorical bins of a hash-space;

applying a hash function to the production space ID and the features of the production event as transformed to retrieve likelihood coefficients for the transformed features of the production event and a standard candle for the production space ID;

calculating an anomaly score;

when the anomaly score represents a detected anomaly event, accessing history associated with the production space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior events for the production space ID; and

invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected;

wherein the likelihood coefficients had been calculated by space ID and a standard candle and mapped into the hash-space using a loosely supervised machine learning of observed features in security-related events using a loss function analyzer and recording the standard candle.

18. A non-transitory computer readable storage medium impressed with computer program instructions to detect anomalies in an event stream of security-related events of one or more organizations, which computer program instructions, when executed on a processor, implement a method of:

receiving a security-related event that includes a space identifier (ID) and features with certain values;

transforming feature-value pairs into categorical bins of a hash-space;

applying a hash function to a combination of the space ID and the transformed feature-value pairs to retrieve likelihood coefficients for the transformed feature-value pairs and a standard candle for the space ID;

scoring the likelihood coefficients and any transformed feature-value pairs that do not have likelihood coefficients because the transformed feature-value pairs that do not have likelihood coefficients were not previously observed for the space ID, in combination with evaluating the standard candle, to produce an anomaly score;

when the anomaly score represents a detected anomaly event, accessing history associated with the space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior security-related events for the space ID; and

invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected.

19. A system including one or more processors coupled to memory, the memory loaded with computer instructions, which computer instructions, when executed on the processors, implement a method of:

evaluating a plurality of production events with production space IDs, including for a production event:

transforming features of the production event into categorical bins of a hash-space;

applying a hash function to the production space ID and the features of the production event as transformed to retrieve likelihood coefficients for the transformed features of the production event and a standard candle for the production space ID;

calculating an anomaly score;

when the anomaly score represents a detected anomaly event, accessing history associated with the production space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior events for the production space ID; and

invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected;

wherein the likelihood coefficients had been calculated by space ID and a standard candle and mapped into the hash-space using a loosely supervised machine learning of observed features in security-related events using a loss function analyzer and recording the standard candle.

20. A system including one or more processors coupled to memory, the memory loaded with computer instructions, which computer instructions, when executed on the processors, implement a method of:

receiving a security-related event that includes a space identifier (ID) and features with certain values;

transforming feature-value pairs into categorical bins of a hash-space;

applying a hash function to a combination of the space ID and the transformed feature-value pairs to retrieve likelihood coefficients for the transformed feature-value pairs and a standard candle for the space ID;

scoring the likelihood coefficients and any transformed feature-value pairs that do not have likelihood coefficients because the transformed feature-value pairs that do not have likelihood coefficients were not previously observed for the space ID, in combination with evaluating the standard candle, to produce an anomaly score;

when the anomaly score represents a detected anomaly event, accessing history associated with the space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior security-related events for the space ID; and

invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected.