Granted Patent B2

US 12,405,975 · App. 17/216,496 · Granted Sep 2, 2025

Inventors: Mark Edward Johnson (Castle Cove, AU); Thanh Long Duong (Seabrook, AU); Vishal Vishnoi (Redwood City, CA); Balakota Srinivas Vinnakota (Sunnyvale, CA); Tuyen Quang Pham (Springvale, AU); Cong Duy Vu Hoang (Wantima South, AU)

Assignee: Oracle International Corporation

G06F16/3329G06F18/211G06F18/217G06N3/08G06N20/00H04L51/02

Abstract

Claims (62)

1. A method comprising:

obtaining one or more digital datasets for training a machine-learning model;

selecting a plurality of metrics for evaluating performance of the machine-learning model on the one or more digital datasets, each metric of the plurality of metrics being configured based on a specification set associated with the metric, and wherein each metric is configured independently of other metrics included in the plurality of metrics, wherein the plurality of metrics is selected to include at least a size of the machine-learning model, a training time of the machine-learning model, an accuracy of the machine-learning model, a stability of the machine-learning model, a regression error of the machine-learning model, and a confidence score of the machine-learning model;

assigning a first weight to each metric of the plurality of metrics, wherein the first weight specifies an importance of each metric in evaluating the performance of the machine-learning model;

creating a loss function that measures the performance of the machine-learning model based on the plurality of metrics and the first weights assigned to each of the plurality of metrics;

tuning a set of hyperparameters associated with the machine-learning model in order to obtain a minimum value of the loss function across the plurality of metrics, thereby optimizing the machine-learning model for the plurality of metrics,

wherein the tuning comprises:

(i) training the machine-learning model using the one or more digital datasets, wherein the machine-learning model is configured based on a current set of values for the set of hyperparameters, the current set of values being determined for a current iteration of the tuning;

(ii) evaluating, using the loss function, performance of the machine-learning model on the one or more digital datasets, thereby obtaining a current value of the loss function for the current iteration;

(iii) determining whether the machine-learning model is optimized for the plurality of metrics by comparing the current value of the loss function with a value of the loss function obtained in a previous iteration;

(iv) in response to the machine-learning model not being optimized for the plurality of metrics: (a) modifying the current set of values for the set of hyperparameters to generate a modified set of hyperparameters, (b) reconfiguring the machine-learning model with the modified set of hyperparameters, and (c) repeating steps (i)-(iii) using the reconfigured machine-learning model, to determine the minimum value of the loss function; and

(v) in response to the machine-learning model being optimized for the plurality of metrics, providing the machine-learning model configured with a set of optimal values of hyperparameters in an iteration determined to have the minimum value of the loss function, as a trained machine-learning model;

incorporating the trained machine-learning model in a digital assistant chatbot system (DACS) configured for a set of predetermined intents; and

based on an input utterance of a user provided to the DACS, inputting the input utterance to the trained machine-learning model, and outputting, by the trained machine-learning model, an intent for the input utterance from the set of predetermined intents.

2. The method of claim 1 , further comprising:

assigning a second weight to each digital dataset of the one or more digital datasets, wherein the second weight specifies an importance of each digital dataset in training the machine-learning model.

3. The method of claim 1 , further comprising:

establishing one or more constraints based on one or more hyperparameters of the set of hyperparameters,

wherein the machine-learning model satisfies each of the one or more constraints.

4. The method of claim 3 , wherein a first constraint of the one or more constraints corresponds to requiring a model size of the machine-learning model to be less than a threshold model size or requiring a training time of the machine-learning model to be less than a threshold time limit.

5. The method of claim 1 , wherein the machine-learning model is a neural network model, and the set of hyperparameters includes at least a number of layers of the machine-learning model, a learning rate of the machine-learning model, a number of hidden units in each layer of the machine-learning model, and a learning algorithm utilized to train the machine-learning model.

6. The method of claim 1 , wherein the loss function utilizes an asymmetric loss technique.

7. A computing device comprising:

a processor; and

a memory including instructions that, when executed with the processor, cause the computing device to, at least:

obtain one or more digital datasets for training a machine-learning model;

select a plurality of metrics for evaluating performance of the machine-learning model on the one or more digital datasets, each metric of the plurality of metrics being configured based on a specification set associated with the metric, and wherein each metric is configured independently of other metrics included in the plurality of metrics, wherein the plurality of metrics is selected to include at least a size of the machine-learning model, a training time of the machine-learning model, an accuracy of the machine-learning model, a stability of the machine-learning model, a regression error of the machine-learning model, and a confidence score of the machine-learning model;

assign a first weight to each metric of the plurality of metrics, wherein the first weight specifies an importance of each metric in evaluating the performance of the machine-learning model;

create a loss function that measures the performance of the machine-learning model based on the plurality of metrics and the first weights assigned to each of the plurality of metrics;

tune a set of hyperparameters associated with the machine-learning model in order to obtain a minimum value of the loss function across the plurality of metrics, thereby optimizing the machine-learning model for the plurality of metrics,

wherein tuning the set of hyperparameters includes:

(i) training the machine-learning model using the one or more digital datasets, wherein the machine-learning model is configured based on a current set of values for the set of hyperparameters, the current set of values being determined for a current iteration of the tuning;

(ii) evaluating, using the loss function, performance of the machine-learning model on the one or more digital datasets, thereby obtaining a current value of the loss function for the current iteration;

(iii) determining whether the machine-learning model is optimized for the plurality of metrics by comparing the current value of the loss function with a value of the loss function obtained in a previous iteration;

(iv) in response to the machine-learning model not being optimized for the plurality of metrics: (a) modifying the current set of values for the set of hyperparameters to generate a modified set of hyperparameters, (b) reconfiguring the machine-learning model with the modified set of hyperparameters, and (c) repeating steps (i)-(iii) using the reconfigured machine-learning model, to determine the minimum value of the loss function; and

(v) in response to the machine-learning model being optimized for the plurality of metrics, providing the machine-learning model configured with a set of optimal values of hyperparameters in an iteration determined to have the minimum value of the loss function, as a trained machine-learning model;

incorporate the trained machine-learning model in a digital assistant chatbot system (DACS) configured for a set of predetermined intents; and

based on an input utterance of a user provided to the DACS, provide, as an input, the input utterance to the trained machine-learning model, and obtain, as an output of the trained machine-learning model, an intent for the input utterance from the set of predetermined intents.

8. The computing device of claim 7 , wherein the processor is further configured to assign a second weight to each digital dataset of the one or more digital datasets, wherein the second weight specifies an importance of each digital dataset in training the machine-learning model.

9. The computing device of claim 7 , wherein the processor is further configured to establish one or more constraints based on one or more hyperparameters of the set of hyperparameters, and

wherein the trained machine-learning model satisfies each of the one or more constraints.

10. The computing device of claim 9 , wherein a first constraint of the one or more constraints corresponds to requiring a model size of the machine-learning model to be less than a threshold model size or requiring a training time of the machine-learning model to be less than a threshold time limit.

11. The computing device of claim 7 , wherein the machine-learning model is a neural network model, and the set of hyperparameters includes at least a number of layers of the machine-learning model, a learning rate of the machine-learning model, a number of hidden units in each layer of the machine-learning model, and a learning algorithm utilized to train the machine-learning model.

12. A non-transitory computer readable medium storing specific computer-executable instructions that, when executed by a processor, cause a computer system to at least:

obtain one or more digital datasets for training a machine-learning model;

select a plurality of metrics for evaluating performance of the machine-learning model on the one or more digital datasets, each metric of the plurality of metrics being configured based on a specification set associated with the metric, and wherein each metric is configured independently of other metrics included in the plurality of metrics, wherein the plurality of metrics is selected to include at least a size of the machine-learning model, a training time of the machine-learning model, an accuracy of the machine-learning model, a stability of the machine-learning model, a regression error of the machine-learning model, and a confidence score of the machine-learning model;

assign a first weight to each metric of the plurality of metrics, wherein the first weight specifies an importance of each metric in evaluating the performance of the machine-learning model;

create a loss function that measures the performance of the machine-learning model based on the plurality of metrics and the first weights assigned to each of the plurality of metrics;

tune a set of hyperparameters associated with the machine-learning model in order to obtain a minimum value of the loss function across the plurality of metrics, thereby optimizing the machine-learning model for the plurality of metrics,

wherein tuning the set of hyperparameters includes:

(i) training the machine-learning model using the one or more digital datasets, wherein the machine-learning model is configured based on a current set of values for the set of hyperparameters, the current set of values being determined for a current iteration of the tuning;

(ii) evaluating, using the loss function, performance of the machine-learning model on the one or more digital datasets, thereby obtaining a current value of the loss function for the current iteration;

(iii) determining whether the machine-learning model is optimized for the plurality of metrics by comparing the current value of the loss function with a value of the loss function obtained in a previous iteration;

(iv) in response to the machine-learning model not being optimized for the plurality of metrics: (a) modifying the current set of values for the set of hyperparameters to generate a modified set of hyperparameters, (b) reconfiguring the machine-learning model with the modified set of hyperparameters, and (c) repeating steps (i)-(iii) using the reconfigured machine-learning model, to determine the minimum value of the loss function; and

(v) in response to the machine-learning model being optimized for the plurality of metrics, providing the machine-learning model configured with a set of optimal values of hyperparameters in an iteration determined to have the minimum value of the loss function, as a trained machine-learning model;

incorporate the trained machine-learning model in a digital assistant chatbot system (DACS) configured for a set of predetermined intents; and

based on an input utterance of a user provided to the DACS, provide, as an input, the input utterance to the trained machine-learning model, and obtain, as an output of the trained machine-learning, an intent for the input utterance from the set of predetermined intents.

13. The non-transitory computer readable medium of claim 12 , wherein the computer system is further configured to assign a second weight to each digital dataset of the one or more digital datasets, wherein the second weight specifies an importance of each digital dataset in training the machine-learning model.

14. The non-transitory computer readable medium of claim 12 , wherein the computer system is further configured to establish one or more constraints based on one or more hyperparameters of the set of hyperparameters, and

wherein the trained machine-learning model satisfies each of the one or more constraints.

15. The non-transitory computer readable medium of claim 14 , wherein a first constraint of the one or more constraints corresponds to requiring a model size of the machine-learning model to be less than a threshold model size or requiring a training time of the machine-learning model to be less than a threshold time limit.

16. The non-transitory computer readable medium of claim 12 , wherein the machine-learning model is a neural network model, and the set of hyperparameters includes at least a number of layers of the machine-learning model, a learning rate of the machine-learning model, a number of hidden units in each layer of the machine-learning model, and a learning algorithm utilized to train the machine-learning model.

Assignments (1)

Continuity (3)

References Cited (40)