Detecting behavioral anomaly in machine learned rule sets

Inventor: Babak Hodjat (Dublin, CA)

Assignee: Cognizant Technology Solutions U.S. Corporation

H04L63/1425 G06N3/126 H04L41/16 G06N5/025 G06N5/045

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,336,672
App. No.: 16/934,935
Granted: May 17, 2022
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Roughly described, anomalous behavior of a machine-learned computer-implemented individual can be detected while operating in a production environment. A population of individuals is represented in a computer storage medium, each individual identifying actions to assert in dependence upon input data. As part of machine learning, the individuals are tested against samples of training data and the actions they assert are recorded in a behavior repository. The behavior of an individual is characterized from the observations recorded during training. In a production environment, the individuals are operated by applying production input data, and the production behavior of the individual is observed and compared to the behavior of the individual represented in the behavior repository. A determination is made from the comparison of whether the individual's production behavior during operation is anomalous.

Claims (33)

1. A method for detecting anomalous behavior of a computer-implemented machine learned rule-based individual operating in a production environment for controlling a system, the individual defining actions in accordance with one or more machine learned rules to assert in dependence upon input data, the method comprising:

generating by a computer in a training environment, a machine-learned rule set, wherein the machine-learned rule set is trained on training data having known patterns of data, the machine-learned rule set defining one or more specific actions to be taken when input data meets one or more specific defined conditions and further wherein there are expected outcomes from the application of the one or more specific actions;

storing in a behavior repository the machine learned rule set with expected outcomes when applied to the training data for the individual as a baseline behavior vector;

in a production environment, applying the machine-learned rule set to production data having unknown patterns of data;

monitoring production outcomes from the application of the machine-learned rule set to the production data and comparing production outcomes to the expected outcomes from the behavior repository to determine whether the machine-learned rules set's behavior during production is anomalous as compared to the machine learned rule set's behavior during training; and

replacing the baseline behavior vector with a production behavior vector when non-anomalous production outcomes are determined from the comparison and controlling a system using non-anomalous production outcomes;

wherein responsive to determining anomalous behavior, performing at least one member of the group consisting of:

blocking one or more specific actions of the machine-learned rule set which caused the anomalous behavior;

logging one or more specific actions of the machine-learned rule set which caused the anomalous behavior;

removing the machine-learned rule set which caused the anomalous behavior from the production environment;

raising an alert to cause independent validation of one or more specific actions one or more specific actions of the machine-learned rule set which caused the anomalous behavior; and

raising an alert to cause ongoing monitoring of the machine-learned rule set which caused the anomalous behavior.

2. The method of claim 1 , wherein the machine learning is based on a genetic algorithm.

3. The method of claim 1 , wherein the controlled system is selected from the group consisting of a mechanical system, an electrical system and a computer system.

4. The method of claim 1 , further comprising, responsive to determining that the machine-learned rule set's behavior is not anomalous during production, storing the production outcomes in the behavior repository as a production behavior vector.

5. The method of claim 1 , wherein comparing production outcomes to the expected outcomes includes determining a distance in vector space between the baseline behavior vector and a production behavior vector.

6. A non-transitory computer readable storage medium having stored therein instructions for detecting anomalous behavior of a computer-implemented machine learned rule-based individual operating in a production environment for controlling a system, the individual defining actions in accordance with one or more machine learned rules to assert in dependence upon input data, the instructions comprising:

generating in a training environment, a machine-learned rule set, wherein the machine-learned rule set is trained on training data having known patterns of data, the machine-learned rule set defining one or more specific actions to be taken when input data meets one or more specific defined conditions and further wherein there are expected outcomes from the application of the one or more specific actions;