DISTRIBUTED AND REDUNDANT MACHINE LEARNING QUALITY MANAGEMENT

1 . A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors in a computing system effectuate operations to execute quality management of modelling methods for implementation of a machine learning design in an object-oriented modeling (OOM) framework, the operations comprising:

writing, with the computing system, modelling-object classes using object-oriented modelling of the modelling methods, the modelling-object classes being members of a set of class libraries;

writing, with the computing system, quality-management classes using object-oriented modelling of quality management, the quality-management classes being members of the set of class libraries;

scanning, with the computing system, modelling-object classes in the set of class libraries to determine modelling-object class definition information;

scanning, with the computing system, quality-management classes in the set of class libraries to determine quality-management class definition information;

using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce object manipulation functions that allow a quality management system to access methods and attributes of modelling-object classes to manipulate objects of the modelling-object classes; and

using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce access to the object manipulation functions.

2 . The medium of claim 1 , wherein:

executing quality management comprises executing a process that integrates raw data ingestion, manipulation, transformation, composition, and storage for building artificial intelligence models.

3 . The medium of claim 1 , wherein:

the modeled quality management comprises management of extract, transform, and load (ETL) phases of a machine learning model designed in the OOM framework.

4 . The medium of claim 1 , wherein:

the modeled quality management comprises reporting of model performance of a machine learning model designed in the OOM framework.

5 . The medium of claim 4 , wherein:

model performance is measured by recall, precision, or F1 score.

6 . The medium of claim 1 , wherein:

the modeled quality management comprises data quality monitoring (DQM).

7 . The medium of claim 6 , wherein:

DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, or datatype changes.

8 . The medium of claim 1 , wherein:

the modeled quality management comprises model quality monitoring (MQM) of a machine learning model designed in the OOM framework.

9 . The medium of claim 8 , wherein:

MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric.

10 . The medium of claim 1 , wherein:

the modeled quality management comprises score quality monitoring (SQM) of a machine learning model designed in the OOM framework.

11 . The medium of claim 10 , wherein:

SQM comprises performing a model hypothesis test.

12 . The medium of claim 10 , wherein:

SQM comprises computing a lift table or a decile table.

13 . The medium of claim 1 , wherein:

the modeled quality management comprises label quality monitoring (LQM) of a machine learning model designed in the OOM framework.

14 . The medium of claim 13 , wherein:

LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources.

15 . The medium of claim 1 , wherein:

the modeled quality management comprises bias quality monitoring (BQM) of a machine learning model designed in the OOM framework.

16 . The medium of claim 15 , wherein

BQM comprises detecting information bias, selection bias, or confounding by the machine learning model designed in the OOM framework.

17 . The medium of claim 1 , wherein:

the modeled quality management comprises privacy quality monitoring (PQM) of a machine learning model designed in the OOM framework.

18 . The medium of claim 1 , wherein:

the modeled quality management comprises data quality monitoring (DQM) of a machine learning model designed in the object-oriented modeling (OOM) framework;

DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, and datatype changes;

the modeled quality management comprises model quality monitoring (MQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;

MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric;

the model-based metric is indicative of an F1 score, accuracy, precision, mean error, media error, distance measure, or recall;

the modeled quality management comprises score quality monitoring (SQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;

SQM comprises performing a model hypothesis test and computing a lift table and a decile table based on predicted probability of positive class membership, based on a cumulative distribution function of positive cases;

the model hypothesis test comprises a Welch's t-test, Kolmogorov-Smirnov test, or a Mann-Whitney U-test;

the modeled quality management comprises label quality monitoring (LQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;

LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources;

the modeled quality management comprises bias quality monitoring (BQM) of the machine learning model designed in the object-oriented modeling (OOM) framework;

BQM comprises detecting information bias, selection bias, and confounding by the machine learning model designed in the object-oriented modeling (OOM) framework;

the modeled quality management comprises privacy quality monitoring (PQM) of the machine learning model designed in the OOM framework.

19 . The medium of claim 1 , wherein:

the modeled quality management comprises a process to determine data source reliability.

20 . The medium of claim 1 , wherein:

an attribute of a quality-management object in one of the quality-management classes comprise means for characterizing quality with the attribute of the quality-management object.