Granted Patent B2

US 12,639,617 · App. 17/150,657 · Granted May 26, 2026

Inventors: Raghav Batta (Ossining, NY); Michael Elton Nidd (Zurich, CH); Larisa Shwartz (Greenwich, CT); Jinho Hwang (Ossining, NY); Harshit Kumar (Delhi, IN)

Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION

G06N20/00G06F16/2358G06Q10/0635

Abstract

Claims (102)

1 . A system, comprising:

a processor that executes computer executable components stored in memory, the computer executable components comprising:

an extraction component that identifies change events in historic operational data that induced one or more incidents in a computing environment;

a trainer component that concurrently trains a plurality of predictive models based on steps comprising:

generating a vectorized model training dataset from a ground truth dataset comprising the change events and the one or more incidents, wherein the generation of the vectorized model training data from the ground truth dataset comprises transformation of text features in the ground truth dataset to vectors using word embeddings;

using one or more explicit or inexplicit change-incidents linkages to tag problematic change events or corresponding incidents to tag changes that closed as failed or that induced one or more incidents before they were closed;

training a sequence model employing the vectorized model training dataset;

generating return sequences representing time steps corresponding to the vectorized model training dataset and the ground truth dataset;

concatenating risk feature data to a return sequence layer of a sequence model to generate combined vector representations;

concurrently training multiple ones of the plurality of predictive models via employing the combined vector representations; and

assigning a change risk assessment score to a defined change, wherein the defined change is a change proposed in a computing environment; and

selecting a predictive model of the plurality of predictive models that performs relatively better than others of the plurality of predictive models based on hold out set evaluations, wherein the plurality of predictive models comprise a machine learning or artificial intelligence model that includes a forecast model, a classification model, an outliers model, a time series model or a clustering model;

an assessment component that:

provides the change risk assessment score to an expert entity to verify accuracy of the change risk assessment score; and

employs, based on receiving expert entity feedback, the trainer component to finetune the selected predictive model by modifying one or more features or corresponding weights used by the selected predictive model; and

explanation component that employs one or more model explainability techniques to derive features that explain the change risk assessment score and provides the change risk assessment score generated by the assessment component.

2 . The system of claim 1 , wherein the risk feature data is obtained from at least one of an expert entity or a deployment topology associated with the defined change or the computing environment to generate combined vector representations.

3 . The system of claim 1 , wherein the explanation component also identifies features in change data representing the defined change and further identifies the features in change events data representing the change events to explain the change risk assessment score.

4 . The system of claim 1 , wherein the assessment component employs the model to assign or alter the change risk assessment score based on the feedback from the expert entity, thereby reducing false positive change risk assessment scores output by the system, improving accuracy of the change risk assessment score, or reducing operational risk associated with one or more computing resources of the computing environment.

5 . A computer-implemented method, comprising:

identifying, by a system operatively coupled to a processor, change events in historic operational data that induced one or more incidents in a computing environment;

concurrently training a plurality of predictive models based on steps comprising:

generating a vectorized model training dataset from a ground truth dataset comprising the change events and the one or more incidents, wherein the generation of the vectorized model training data from the ground truth dataset comprises transformation of text features in the ground truth dataset to vectors using word embeddings;

using one or more explicit or inexplicit change-incidents linkages to tag problematic change events or corresponding incidents to tag changes that closed as failed or that induced one or more incidents before they were closed;

training a sequence model employing the vectorized model training dataset;

generating return sequences representing time steps corresponding to the vectorized model training dataset and the ground truth dataset;

concatenating risk feature data to a return sequence layer of a sequence model to generate combined vector representations;

concurrently training the plurality of predictive models via employing the combined vector representations; and

assigning a change risk assessment score to a defined change, wherein the defined change is a change proposed in a computing environment;

selecting, by the system, a predictive model of the plurality of predictive models that performs relatively better than others of the plurality of predictive models based on hold out set evaluations, wherein the plurality of predictive models comprise a machine learning or artificial intelligence model that includes a forecast model, a classification model, an outliers model, a time series model or a clustering model;

providing, by the system, the change risk assessment score to an expert entity to verify accuracy of the change risk assessment score;

employing, by the system, based on receiving expert entity feedback, the trainer component to finetune the selected predictive model by modifying one or more features or corresponding weights used by the selected predictive model; and

employing, by the system, one or more model explainability techniques to derive features that explain the change risk assessment score and provides the change risk assessment score generated by the assessment component.

6 . The computer-implemented method of claim 5 , wherein the return sequences with risk feature data is obtained from at least one of the expert entity or a deployment topology associated with the defined change or the computing environment to generate the combined vector representations.

7 . The computer-implemented method of claim 5 , further comprising:

identifying, by the system, features in change data representing the defined change; and

identifying, by the system, the features in change events data representing the change events to explain the change risk assessment score.

8 . The computer-implemented method of claim 5 , further comprising:

employing, by the system, the plurality of predictive models to assign or alter the change risk assessment score based on the feedback from the expert entity, thereby reducing false positive change risk assessment scores output by the system, improving accuracy of the change risk assessment score, or reducing operational risk associated with one or more computing resources of the computing environment.

9 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

concurrently train a plurality of predictive models based on steps comprising:

generation of a vectorized model training dataset from a ground truth dataset comprising the change events and the one or more incidents, wherein the generation of the vectorized model training data from the ground truth dataset comprises transformation of text features in the ground truth dataset to vectors using word embeddings;

a using of one or more explicit or inexplicit change-incidents linkages to tag problematic change events or corresponding incidents to tag changes that closed as failed or that induced one or more incidents before they were closed;

a training of a sequence model employing the vectorized model training dataset;

generation of return sequences representing time steps corresponding to the vectorized model training dataset and the ground truth dataset;

concatenation of risk feature data to a return sequence layer of a sequence model to generate combined vector representations;

a concurrently training of the plurality of predictive models via employing the combined vector representations; and

assignment of a change risk assessment score to a defined change, wherein the defined change is a change proposed in a computing environment;

select a predictive model of the plurality of predictive models that performs relatively better than others of the plurality of predictive models based on hold out set evaluations, wherein the plurality of predictive models comprise a machine learning or artificial intelligence model that includes a forecast model, a classification model, an outliers model, a time series model or a clustering model;

provide the change risk assessment score to an expert entity to verify accuracy of the change risk assessment score;

employ based on receiving expert entity feedback, the trainer component to finetune the selected predictive model by modifying one or more features or corresponding weights used by the selected predictive model; and

employ one or more model explainability techniques to derive features that explain the change risk assessment score and provides the change risk assessment score generated by the assessment component.

10 . The computer program product of claim 9 , wherein the program instructions are further executable by the processor to cause the processor to:

generate a vectorized model training dataset from a ground truth dataset comprising the change events and the one or more incidents; and

train, using the vectorized model training dataset, a sequence model to generate return sequences representing time steps corresponding to the vectorized model training dataset.

11 . The computer program product of claim 10 , wherein the program instructions are further executable by the processor to cause the processor to:

combine the return sequences with risk feature data obtained from at least one of the expert entity or a deployment topology associated with the defined change or the computing environment to generate combined vector representations; and

train, using the combined vector representations, the model to assign the change risk assessment score to the defined change.

12 . The computer program product of claim 9 , wherein the program instructions are further executable by the processor to cause the processor to:

identify features in change data representing the defined change; and

identify the features in change events data representing the change events to explain the change risk assessment score.

13 . The computer program product of claim 9 , wherein the program instructions are further executable by the processor to cause the processor to:

employ the model to assign or alter the change risk assessment score based on the feedback from the expert entity, thereby reducing false positive change risk assessment scores output by the processor, improving accuracy of the change risk assessment score, or reducing operational risk associated with one or more computing resources of the computing environment.

14 . A system, comprising:

a processor that executes computer executable components stored in memory, the computer executable components comprising:

a trainer component that concurrently trains a plurality of predictive models based on steps comprising:

generating a vectorized model training dataset from a ground truth dataset comprising the change events and the one or more incidents, wherein the generation of the vectorized model training data from the ground truth dataset comprises transformation of text features in the ground truth dataset to vectors using word embeddings;

using one or more explicit or inexplicit change-incidents linkages to tag problematic change events or corresponding incidents to tag changes that closed as failed or that induced one or more incidents before they were closed;

training a sequence model employing the vectorized model training dataset;

generating return sequences representing time steps corresponding to the vectorized model training dataset and the ground truth dataset;

concatenating risk feature data to a return sequence layer of a sequence model to generate combined vector representations;

concurrently training the plurality of predictive models via employing the combined vector representations; and

assigning a change risk assessment score to a defined change, wherein the defined change is a change proposed in a computing environment; and

an assessment component that:

provides the change risk assessment score to an expert entity to verify accuracy of the change risk assessment score; and

employs, based on receiving expert entity feedback, the trainer component to finetune the selected predictive model by modifying one or more features or corresponding weights used by the selected predictive model; and

an explanation component that:

displays, via a graphical user interface (GUI), over the Internet, an explanation of the change risk assessment score using change data that represents at least one of the change events and that has one or more features of the defined change; and

employs one or more model explainability techniques to derive features that explain the change risk assessment score and provides the change risk assessment score generated by the assessment component, wherein the GUI is located at a first location distinct from a second location of the computing environment, wherein the plurality of predictive models comprise a machine learning or artificial intelligence model that includes a forecast model, a classification model, an outliers model, a time series model or a clustering model.

15 . The system of claim 14 , wherein the computer executable components further comprise:

an extraction component that identifies at least one of the change events or the one or more incidents in historic operational data of the computing environment.

16 . The system of claim 14 , wherein the trainer component combines the return sequences with risk feature data obtained from at least one of the expert entity or a deployment topology associated with the defined change or the computing environment to generate the combined vector representations, and wherein the trainer component further uses the combined vector representations to train the predictive model to assign the change risk assessment score to the defined change.

17 . The system of claim 14 , wherein the assessment component employs the predictive model to assign or alter the change risk assessment score based on feedback from the expert entity, thereby reducing false positive change risk assessment scores output by the system, improving accuracy of the change risk assessment score, or reducing operational risk associated with one or more computing resources of the computing environment.

18 . A computer-implemented method, comprising:

generating, by a system operatively coupled to a processor, vectorized model training dataset from a ground truth dataset comprising change events and one or more incidents;

concurrently training a plurality of predictive models based on steps comprising:

generating a vectorized model training dataset from a ground truth dataset comprising the change events and the one or more incidents, wherein the generation of the vectorized model training data from the ground truth dataset comprises transformation of text features in the ground truth dataset to vectors using word embeddings;

using one or more explicit or inexplicit change-incidents linkages to tag problematic change events or corresponding incidents to tag changes that closed as failed or that induced one or more incidents before they were closed;

training a sequence model employing the vectorized model training dataset;

generating return sequences representing time steps corresponding to the vectorized model training dataset and the ground truth dataset;

concatenating risk feature data to a return sequence layer of a sequence model to generate combined vector representations;

concurrently training the plurality of predictive models via employing the combined vector representations; and

assigning a change risk assessment score to a defined change, wherein the defined change is a change proposed in a computing environment;

providing, by the system, the change risk assessment score to an expert entity to verify accuracy of the change risk assessment score;

employing, by the system, based on receiving expert entity feedback, the trainer component to finetune the selected predictive model by modifying one or more features or corresponding weights used by the selected predictive model; and

displaying, by the system, via an application programming interface, an explanation of the change risk assessment score using change data associated with a change in source code or configuration changes and that represents at least one of the change events and that has one or more features of the defined change, employing, by the system, one or more model explainability techniques to derive features that explain the change risk assessment score and provides the change risk assessment score generated by the assessment component, wherein the plurality of predictive models comprise a machine learning or artificial intelligence model that includes a forecast model, a classification model, an outliers model, a time series model or a clustering model.

19 . The computer-implemented method of claim 18 , further comprising:

identifying, by the system, at least one of the change events or the one or more incidents in historic operational data of the computing environment.

20 . The computer-implemented method of claim 18 , further comprising:

combining, by the system, the return sequences with risk feature data obtained from at least one of expert entity or a deployment topology associated with the defined change or the computing environment to generate combined vector representations.

21 . The computer-implemented method of claim 18 , further comprising:

employing, by the system, the predictive model to assign or alter the change risk assessment score based on feedback from an expert entity, thereby reducing false positive change risk assessment scores output by the system, improving accuracy of the change risk assessment score, or reducing operational risk associated with one or more computing resources of the computing environment.

Assignments (1)

Continuity (1)

References Cited (19)