Granted Patent B2

US 12,725,076 · App. 17/506,753 · Granted Sep 1, 2026

Inventors: Sahil Suneja (Ossining, NY); Yufan Zhuang (New York, NY); Yunhui Zheng (Chappaqua, NY); Alessandro Morari (New York, NY); Jim Alain Laredo (Katonah, NY)

Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION

G06N20/00

Abstract

Claims (72)

1 . A system, comprising:

a memory that stores computer executable components; and

a processor, operably coupled to the memory, and that executes the computer executable components stored in the memory, wherein the computer executable components comprise:

a model introspection component that analyzes artificial intelligence model learning behavior for a code understanding task;

an extraction component that extracts one or more code complexity metrics for a plurality of source code samples included in a testing dataset;

a testing data subset component that generates a first testing data subset and a second testing data subset by grouping the plurality of source code samples based on a performance metric that evaluates prediction correctness of the artificial intelligence model, wherein the first testing data subset comprises source code samples associated with a true positive result of the artificial intelligence model, and wherein the second testing data subset comprises source code samples associated with a false negative result of the artificial intelligence model;

a distribution component that determines, based on the one or more code complexity metrics, a first distribution of the source code samples within the first testing data subset and a second distribution of the source code samples within the second testing data subset; and

a comparison component that compares the first distribution with the second distribution, wherein the model introspection component determines, based on the comparison, aspects of the source code samples that the artificial intelligence model has learned and aspects of the source code samples associated with prediction errors, and generates machine-readable guidance data and causes retraining of the artificial intelligence model based on the identified aspects such that the retrained artificial intelligence model reduces prediction errors associated with the identified aspects of the source code samples.

2 . The system of claim 1 , wherein the performance metric comprises prediction correctness of the artificial intelligence model, and wherein the testing data subset component is configured to group the plurality of source code samples into testing data subsets based on whether corresponding predictions of the artificial intelligence model are correct or incorrect.

3 . The system of claim 1 , wherein the distribution component determines a distribution of the plurality of source code samples within a plurality of testing data subsets based on the one or more code complexity metrics.

4 . The system of claim 3 , further comprising:

a comparison component that compares a first distribution of source code samples associated with a first testing data subset from the plurality of testing data subsets with a second distribution of source code samples associated with a second testing data subset from the plurality of testing data subsets;

wherein the testing data subset component groups the plurality of source code samples based on quantitative performance metrics derived from artificial intelligence model predictions utilizing prediction confidence scores and error classification;

wherein the extraction component extracts a plurality of code complexity metrics from the source code samples, including at least one of: cyclomatic complexity, Halstead complexity, maintainability index, or control flow complexity; and

wherein the comparison component generates a statistical distribution model of the extracted code complexity metrics across the testing data subsets and identifies outlier patterns and complexity thresholds that affect artificial intelligence model prediction accuracy.

5 . A system, comprising:

a memory that stores computer executable components; and

a processor, operably coupled to the memory, and that executes the computer executable components stored in the memory, wherein the computer executable components comprise:

a model introspection component that deduces learning behavior of an artificial intelligence model for a code understanding task by:

extracting one or more code complexity metrics from a plurality of source code samples included in a testing dataset;

programmatically generating a plurality of testing data subsets by grouping the plurality of source code samples based on a performance metric that evaluates prediction outputs of the artificial intelligence model;

executing the artificial intelligence model on each of the testing data subsets to obtain per-subset prediction outputs;

partitioning the testing data subsets into (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs;

determining, based on the one or more code complexity metrics, distributions of the source code samples within the subsets associated with correct predictions and the subsets associated with incorrect predictions;

computing comparative differences between the distributions;

identifying, based on the comparative distributions, complexity thresholds and structural code features correlated with prediction errors of the artificial intelligence model; and

generating machine-readable guidance data that specifies modifications to training data or model parameters, and automatically retraining the artificial intelligence model using the identified complexity thresholds and structural code features such that the retrained artificial intelligence model is configured to reduce or eliminate prediction errors for source code samples exhibiting the identified characteristics.

6 . The system of claim 5 , wherein the performance metric comprises prediction correctness of the artificial intelligence model, and wherein the testing data subsets comprise (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs.

7 . The system of claim 5 , wherein a distribution component determines a distribution of the plurality of source code samples within the plurality of testing data subsets based on the one or more code complexity metrics.

8 . The system of claim 7 , further comprising:

a comparison component that compares a first distribution of source code samples associated with a first testing data subset from the plurality of testing data subsets with a second distribution of source code samples associated with a second testing data subset from the plurality of testing data subsets.

9 . A computer-implemented method, comprising:

extracting, by a processor, one or more code complexity metrics from a plurality of source code samples included in a testing dataset;

generating, by the processor, a plurality of testing data subsets by grouping the plurality of source code samples based on a performance metric that evaluates prediction outputs of an artificial intelligence model;

executing, by the processor, the artificial intelligence model on each of the testing data subsets to obtain per-subset prediction outputs;

partitioning, by the processor, the testing data subsets into (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs;

determining, by the processor and based on the one or more code complexity metrics, distributions of the source code samples within the subsets associated with correct predictions and the subsets associated with incorrect predictions;

computing, by the processor, comparative differences between the distributions;

identifying, by the processor and based on the comparative differences, complexity thresholds and structural code features correlated with prediction errors of the artificial intelligence model; and

generating, by the processor, machine-readable guidance data that specifies modifications to training data or model parameters, and retraining, by the processor, the artificial intelligence model based on the identified complexity thresholds and structural code features such that the retrained artificial intelligence model reduces prediction errors associated with the identified characteristics.

10 . The computer-implemented method of claim 9 , wherein the performance metric comprises prediction correctness of the artificial intelligence model, and wherein generating the plurality of testing data subsets comprises grouping the plurality of source code samples into (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs.

11 . The computer-implemented method of claim 9 , further comprising

determining, by the processor, a distribution of the plurality of source code samples within the plurality of testing data subsets based on the one or more code complexity metrics.

12 . The computer-implemented method of claim 11 , further comprising:

comparing, by the processor, a first distribution of source code samples associated with a first testing data subset from the plurality of testing data subsets with a second distribution of source code samples associated with a second testing data subset from the plurality of testing data subsets.

13 . A computer-implemented method, comprising:

extracting, by a processor, one or more code complexity metrics from a plurality of source code samples included in a testing dataset;

generating, by the processor, a plurality of testing data subsets by grouping the plurality of source code samples based on a performance metric that evaluates prediction outputs of an artificial intelligence model;

executing, by the processor, the artificial intelligence model on each of the testing data subsets to obtain per-subset prediction outputs;

partitioning, by the processor, the testing data subsets into (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs;

analyzing, by the processor, performance differences between the testing data subsets by determining, based on the one or more code complexity metrics, distributions of the source code samples within the subsets associated with correct predictions and the subsets associated with incorrect predictions and computing comparative differences between the distributions;

deducing, by the processor and from the comparative differences, aspects of learning behavior of the artificial intelligence model, including identification of complexity thresholds and structural code features correlated with systematic prediction errors; and

generating, by the processor, machine-readable guidance data specifying modifications to training data or model parameters, and retraining, by the processor, the artificial intelligence model based on the identified complexity thresholds and structural code features such that the retrained artificial intelligence model reduces prediction errors associated with the identified characteristics.

14 . The computer-implemented method of claim 13 , wherein the performance metric comprises prediction correctness of the artificial intelligence model, and wherein generating the plurality of testing data subsets comprises grouping the plurality of source code samples into (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs.

15 . The computer-implemented method of claim 13 , further comprising

determining, by the processor, a distribution of the plurality of source code samples within the plurality of testing data subsets based on the one or more code complexity metrics.

16 . The computer-implemented method of claim 15 , further comprising:

comparing, by the processor, a first distribution of source code samples associated with a first testing data subset from the plurality of testing data subsets with a second distribution of source code samples associated with a second testing data subset from the plurality of testing data subsets.

17 . A computer program product for code complexity driven model introspection, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

extract one or more code complexity metrics from a plurality of source code samples included in a testing dataset;

generate a plurality of testing data subsets by grouping the plurality of source code samples based on a performance metric that evaluates prediction outputs of an artificial intelligence model;

execute the artificial intelligence model on each of the testing data subsets to obtain per-subset prediction outputs;

partition the testing data subsets into (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs;

determine, based on the one or more code complexity metrics, distributions of the source code samples within the subsets associated with correct predictions and the subsets associated with incorrect predictions;

compute comparative differences between the distributions;

identify, based on the comparative differences, complexity thresholds and structural code features correlated with prediction errors of the artificial intelligence model; and

generate machine-readable guidance data specifying modifications to training data or model parameters, and retrain the artificial intelligence model based on the identified complexity thresholds and structural code features such that the retrained artificial intelligence model reduces prediction errors associated with the identified characteristics.

18 . The computer program product of claim 17 , wherein the performance metric comprises prediction correctness of the artificial intelligence model, and wherein generating the plurality of testing data subsets comprises grouping the plurality of source code samples into (i) subsets associated with correct predictions and (ii) subsets associated with incorrect predictions based on the per-subset prediction outputs.

19 . The computer program product of claim 17 , wherein the program instructions further cause the processor to:

determine, by the processor, a distribution of the plurality of source code samples within the plurality of testing data subsets based on the one or more code complexity metrics.

20 . The computer program product of claim 19 , wherein the program instructions further cause the processor to:

compare, by the processor, a first distribution of source code samples associated with a first testing data subset from the plurality of testing data subsets with a second distribution of source code samples associated with a second testing data subset from the plurality of testing data subsets.

Assignments (1)

Continuity (1)

References Cited (124)