Granted Patent B2

US 12,632,425 · App. 18/104,586 · Granted May 19, 2026

Inventors: Kripa Kanchana Sivakumar (Seattle, WA); Mayoor Rao (San Jose, CA); John Backof (Tiburon, CA); Andrew Ioannou (San Francisco, CA)

Assignee: Oracle International Corporation

G06F16/215G06F16/258

Abstract

Claims (54)

1 . A method comprising:

accessing, by a machine learning monitoring system, a data monitoring definition that is used to monitor drift or quality of data that is related to one or more machine-learned models, wherein the data monitoring definition includes predefined configuration data and a custom configuration data that is specified by a user through a computing device that is separate from the machine learning monitoring system, wherein the custom configuration data includes custom instructions pertaining to one or more of data reading, metrics generation, or data writing;

based on the data monitoring definition, generating, by the machine learning monitoring system, executable code that comprises a data reading portion, a metrics generation portion, and a data writing portion;

executing, by the machine learning monitoring system, the executable code, wherein executing the executable code comprises:

based on the data reading portion in the executable code, reading a dataset based on location data specified in the data monitoring definition;

based on the metrics generation portion in the executable code, reading the dataset and generating a set of metrics based on the dataset;

based on the data writing portion in the executable code, reading the set of metrics and writing a result that is based on the set of metrics;

wherein the method is performed by one or more computing devices.

2 . The method of claim 1 , wherein the custom configuration data specifies a location to read the dataset.

3 . The method of claim 1 , wherein the custom configuration data specifies a custom data reader that reads the dataset from a specified location.

4 . The method of claim 1 , wherein the custom configuration data specifies a custom data transformer that transforms data from the dataset from one data format to another data format.

5 . The method of claim 1 , wherein the custom configuration data specifies instructions for generating a custom metric based on data in the dataset.

6 . The method of claim 1 , wherein the custom configuration data specifies a custom data writer that is configured to write the result.

7 . The method of claim 1 , further comprising, prior to generating the executable code, analyzing the data monitoring definition for any malformed configuration data.

8 . The method of claim 1 , further comprising, prior to receiving the data monitoring definition:

storing a plurality of data monitoring definition templates that includes a first data monitoring definition template and a second data monitoring definition template that is different than the first data monitoring definition template;

receiving first user input that selects the first data monitoring definition template;

receiving second user input that modifies the first data monitoring definition template to create the data monitoring definition.

9 . One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:

accessing, by a machine learning monitoring system, a data monitoring definition that is used to monitor drift or quality of data that is related to one or more machine-learned models, wherein the data monitoring definition includes predefined configuration data and a custom configuration data that is specified by a user through a computing device that is separate from the machine learning monitoring system, wherein the custom configuration data includes custom instructions pertaining to one or more of data reading, metrics generation, or data writing;

based on the data monitoring definition, generating, by the machine learning monitoring system, executable code that comprises a data reading portion, a metrics generation portion, and a data writing portion;

executing, by the machine learning monitoring system, the executable code, wherein executing the executable code comprises:

based on the data reading portion in the executable code, reading a dataset based on location data specified in the data monitoring definition;

based on the metrics generation portion in the executable code, reading the dataset and generating a set of metrics based on the dataset;

based on the data writing portion in the executable code, reading the set of metrics and writing a result that is based on the set of metrics.

10 . The one or more non-transitory storage media of claim 9 , wherein the custom configuration data specifies a location to read the dataset.

11 . The one or more non-transitory storage media of claim 9 , wherein the custom configuration data specifies a custom data reader that reads the dataset from a specified location.

12 . The one or more non-transitory storage media of claim 9 , wherein the custom configuration data specifies a custom data transformer that transforms data from the dataset from one data format to another data format.

13 . The one or more non-transitory storage media of claim 9 , wherein the custom configuration data specifies instructions for generating a custom metric based on data in the dataset.

14 . The one or more non-transitory storage media of claim 9 , wherein the custom configuration data specifies a custom data writer that is configured to write the result.

15 . The one or more non-transitory storage media of claim 9 , wherein the instructions, when executed by the one or more computing devices, further cause, prior to generating the executable code, analyzing the data monitoring definition for any malformed configuration data.

16 . The one or more non-transitory storage media of claim 9 , wherein the instructions, when executed by the one or more computing devices, further cause, prior to receiving the data monitoring definition:

storing a plurality of data monitoring definition templates that includes a first data monitoring definition template and a second data monitoring definition template that is different than the first data monitoring definition template;

receiving first user input that selects the first data monitoring definition template;

receiving second user input that modifies the first data monitoring definition template to create the data monitoring definition.

17 . A system comprising:

one or more computing devices;

one or more storage media storing instructions which, when executed by the one or more computing devices, cause:

accessing, by a machine learning monitoring system, a data monitoring definition that is used to monitor drift or quality of data that is related to one or more machine-learned models, wherein the data monitoring definition includes predefined configuration data and a custom configuration data that is specified by a user through a computing device that is separate from the machine learning monitoring system, wherein the custom configuration data includes custom instructions pertaining to one or more of data reading, metrics generation, or data writing;

based on the data monitoring definition, generating, by the machine learning monitoring system, executable code that comprises a data reading portion, a metrics generation portion, and a data writing portion;

executing, by the machine learning monitoring system, the executable code, wherein executing the executable code comprises:

based on the data reading portion in the executable code, reading a dataset based on location data specified in the data monitoring definition;

based on the metrics generation portion in the executable code, reading the dataset and generating a set of metrics based on the dataset;

based on the data writing portion in the executable code, reading the set of metrics and writing a result that is based on the set of metrics.

18 . The system of claim 17 , wherein the custom configuration data specifies:

a location to read the dataset, a custom data reader that reads the dataset from a specified location,

a custom data transformer that transforms data from the dataset from one data format to another data format,

instructions for generating a custom metric based on data in the dataset, or

a custom data writer that is configured to write the result.

19 . The system of claim 17 , wherein the instructions, when executed by the one or more computing devices, further cause, prior to generating the executable code, analyzing the data monitoring definition for any malformed configuration data.

20 . The system of claim 17 , wherein the instructions, when executed by the one or more computing devices, further cause, prior to receiving the data monitoring definition:

storing a plurality of data monitoring definition templates that includes a first data monitoring definition template and a second data monitoring definition template that is different than the first data monitoring definition template;

receiving first user input that selects the first data monitoring definition template;

receiving second user input that modifies the first data monitoring definition template to create the data monitoring definition.

Assignments (1)

Continuity (1)

References Cited (15)