Granted Patent B2

US 12,645,959 · App. 17/331,131 · Granted Jun 2, 2026

Inventors: Nham Van Le (Waterloo, CA); Tuan Manh Lai (San Jose, CA); Trung Bui (San Jose, CA); Doo Soon Kim (San Jose, CA)

Assignee: Adobe Inc.

G06N5/04G06N20/00

Abstract

Claims (95)

1 . A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to:

receive, from a client device, a selection of a dataset and a selection of a machine-learning model from a plurality of available machine-learning models associated with a natural language understanding task;

receive, from a client device, a request to instantiate the selected machine-learning model utilizing the selected dataset;

monitor on-demand instances of cloud-based computing devices by utilizing a scheduler comprising computing software comprising software components on one or more computing devices to determine an availability status of the cloud-based computing devices;

determine, within a task attribute interface of a graphical user interface of the client device, a computing hardware configuration in response to one or more client device interactions with computing hardware configuration elements of the task attribute interface;

determine, by utilizing the scheduler to automatically determine availability statuses of cloud-based computing devices, a cloud-based computing device corresponding to a cloud-based storage system based on the selected machine-learning model and one or more attributes associated with the natural language understanding task by selecting, from the on-demand instances of the cloud-based computing devices, an available cloud-based computing device that comprises computing hardware compatible with the one or more attributes associated with the natural language understanding task, the availability status, and the computing hardware configuration;

instantiate the selected machine-learning model at the available cloud-based computing device;

learn parameters of the selected machine-learning model at the cloud-based computing device utilizing the selected dataset;

generate, for display within a graphical user interface of the client device, an interactive dataset visualization comprising a confusion matrix with a plurality of interactive cells mapped to classification data generated by the selected machine-learning model for the selected dataset in connection with the natural language understanding task;

generate, for display and based on a client device interaction with a data sample mapped to an interactive cell from the plurality of interactive cells within the interactive dataset visualization, a modification interface including an initial representation of classification data corresponding to the data sample and user interface tools for modifying classification data of the data sample;

modify the classification data of the data sample mapped to the interactive cell in response to a client device interaction with the user interface tools of the modification interface;

generate, for display within the modification interface, an updated initial representation of modified classification data corresponding to the data sample; and

generate, for display within the graphical user interface of the client device, a modified interactive dataset visualization comprising a modified confusion matrix with a plurality of interactive cells mapped to the modified classification data.

2 . The non-transitory computer readable storage medium as recited in claim 1 , wherein

learning parameters of the selected machine-learning model further comprises instructions that, when executed by at least one processor, cause the computing device to:

access, for the cloud-based computing device, data associated with the select machine-learning model and a training algorithm;

access, according to instantiation data, the dataset; and

utilize the dataset and the training algorithm within the cloud-based computing device to learn parameters of the selected machine-learning model.

3 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by at least one processor, cause the computing device to determine the cloud-based computing device from a plurality of available cloud-based computing devices corresponding to a plurality of cloud-based computing systems.

4 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by at least one processor, cause the computing device to generate, for display at the client device, the interactive dataset visualization by generating classifications for natural language samples in the selected dataset utilizing the selected machine-learning model instantiated at the cloud-based computing device.

5 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by at least one processor, cause the computing device to provide the modification interface by presenting user interface tools comprising a text modification tool and an intent modification tool for modifying classification data of the data sample.

6 . The non-transitory computer readable storage medium as recited in claim 5 , further comprising instructions that, when executed by at least one processor, cause the computing device to:

provide, within the interactive dataset visualization, one or more labels or text associated with an interactive cell within the plurality of interactive cells within the confusion matrix and an intent associated with the interactive cell within the plurality of interactive cells within the confusion matrix;

modify the classification data of the data sample mapped to the interactive cell by modifying the one or more labels or text associated with the interactive cell in response to a client device interaction with the text modification tool;

modify the classification data of the data sample mapped to the interactive cell by modifying the intent associated with the interactive cell in response to a client device interaction with the intent modification tool; and

generate the modified interactive dataset visualization based on the modified interactive cell.

7 . The non-transitory computer readable storage medium as recited in claim 6 , wherein providing the one or more labels or text further comprises instructions that, when executed by at least one processor, cause the computing device to:

determine, for the interactive cell, the intent from metadata associated with selected dataset; and

provide the intent for display within the interactive dataset visualization.

8 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by at least one processor, cause the computing device to:

provide, for display at the client device, a dataset annotation interface comprising tools to annotate data samples in the selected dataset in response to one or more client device interactions with the interactive dataset visualization;

convert the annotated data samples in the selected dataset from an initial representation to an intermediate representation of data in the selected dataset representing the annotated data samples; and

learn parameters of the selected machine-learning model utilizing the intermediate representation of the data in the selected dataset.

9 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by at least one processor, cause the computing device to:

integrate a command-line interface provided by an additional client device;

receive, from the client device via the command-line interface, one or more commands for learning the parameters of the selected machine-learning model; and

store the selected machine-learning model in a database comprising the plurality of available machine-learning models.

10 . A system comprising:

one or more computer memory devices; and

one or more computing devices configured to cause the system to:

receive, from a client device, a request to instantiate a selected machine-learning model from a plurality of available machine-learning models and one or more attributes associated with a natural language understanding task;

monitor on-demand instances of cloud-based computing devices by utilizing a scheduler comprising computing software comprising software components on one or more computing devices to determine an availability status of the cloud-based computing devices;

determine, within a task attribute interface of a graphical user interface of the client device, a computing hardware configuration in response to one or more client device interactions with computing hardware configuration elements of the task attribute interface;

determine, for the selected machine-learning model, a cloud-based computing device corresponding to a cloud-based system by:

monitoring, by utilizing a scheduler, a status of an on-demand cluster of cloud-based computing devices comprising the cloud-based computing device;

determining automatically that the cloud-based computing device is available and comprises computing hardware compatible with the one or more attributes associated with the natural language understanding task; and

selecting, from the on-demand instances of the cloud-based computing devices, an available cloud-based computing device that comprises computing hardware compatible with the one or more attributes associated with the natural language understanding task, the availability status, and the computing hardware configuration;

instantiate the selected machine-learning model at the available cloud-based computing device in connection with a dataset indicated by the client device;

generate, for display within a graphical user interface at the client device, an interactive dataset visualization comprising a confusion matrix with a plurality of interactive cells mapped to classification data generated by the selected machine-learning model for the dataset in connection with the natural language understanding task;

generate, for display and based on a client device interaction with a data sample mapped to an interactive cell from the plurality of interactive cells within the interactive dataset visualization, a modification interface including an initial representation of classification data corresponding to the data sample and user interface tools for modifying classification data of the data sample;

modify the classification data of the data sample mapped to the interactive cell in response to a client device interaction with the user interface tools of the modification interface;

generate, for display within the modification interface, an updated initial representation of modified classification data corresponding to the data sample; and

generate, for display within the graphical user interface of the client device, a modified interactive dataset visualization comprising a modified confusion matrix with a plurality of interactive cells mapped to the modified classification data.

11 . The system as recited in claim 10 , wherein the one or more computing devices are further configured to cause the system to:

receive, from the client device, an indication of the one or more attributes of the natural language understanding task comprising a computing hardware configuration; and

determine automatically that the cloud-based computing device comprises computing hardware corresponding to the computing hardware configuration received from the client device.

12 . The system as recited in claim 10 , wherein the one or more computing devices are further configured to cause the system to generate the interactive dataset visualization by:

generate the confusion matrix by processing the dataset utilizing the selected machine-learning model, the selected machine-learning model comprising a natural language understanding model;

map a plurality of interactive cells in the confusion matrix to data samples from the dataset based on classifications determined by the natural language understanding model; and

provide, for display at the client device, an interactive graphical matrix comprising the confusion matrix with the plurality of interactive cells.

13 . The system as recited in claim 10 , wherein the one or more computing devices are further configured to cause the system to:

convert, in response to detecting one or more modifications to the classification data mapped to the plurality of interactive cells, data samples of the dataset to an intermediate representation of the data samples representing the modified interactive classification data; and

learn parameters of the selected machine-learning model utilizing the intermediate representation of the data samples.

14 . The system as recited in claim 13 , wherein the one or more computing devices are further configured to cause the system to:

receive, from the client device, annotations for the data samples of the dataset;

generate the classification data utilizing the selected machine-learning model by determining classifications for the data samples based on the annotations of the data samples; and

provide, to the client device, the classifications of the data samples within the confusion matrix of the interactive dataset visualization.

15 . The system as recited in claim 14 , wherein the one or more computing devices are further configured to cause the system to:

receive, from the client device, a modified data sample comprising a modified natural language label based on the classifications of the data samples; and

update the parameters of the selected machine-learning model based on the modified data sample.

16 . The system as recited in claim 10 , wherein the one or more computing devices are further configured to cause the system to:

receive, from the client device, a request to download the selected machine-learning model comprising learned parameters; and

provide to the client device for download the selected machine-learning model comprising the learned parameters.

17 . A method comprising:

receiving, from a client device, a request to instantiate a selected machine-learning model from a plurality of available machine-learning models and one or more attributes associated with a natural language understanding task;

monitoring on-demand instances of cloud-based computing devices by utilizing a scheduler comprising computing software comprising software components on one or more computing devices to determine an availability status of the cloud-based computing devices;

determining, within a task attribute interface of a graphical user interface of the client device, a computing hardware configuration in response to one or more client device interactions with computing hardware configuration elements of the task attribute interface;

determining, by utilizing the scheduler to automatically determine availability statuses of cloud-based computing devices for the selected machine-learning model, a cloud-based computing device corresponding to a cloud-based storage system based on the selected machine-learning model and one or more attributes associated with the natural language understanding task by selecting, from the on-demand instances of the cloud-based computing devices, an available cloud-based computing device that comprises computing hardware compatible with the one or more attributes associated with the natural language understanding task, the availability status, and the computing hardware configuration;

instantiating the selected machine-learning model at the available cloud-based computing device in connection with a dataset indicated by the client device; and

generating an interactive dataset visualization comprising interactive-classification data for the dataset by:

generating a confusion matrix based on classifications determined by the selected machine-learning model;

providing, for display within a graphical user interface at the client device, an interactive graphical matrix comprising the confusion matrix with a plurality of interactive cells mapped to the classifications in connection with the natural language understanding task;

generating, for display and based on a client device interaction with a data sample mapped to an interactive cell from the plurality of interactive cells within the interactive dataset visualization, a modification interface including an initial representation of classification data corresponding to the data sample and user interface tools for modifying the classifications of the data sample;

modifying the classifications of the data sample mapped to the interactive cell in response to a client device interaction with the user interface tools of the modification interface;

generating, for display within the modification interface, an updated initial representation of modified classification data corresponding to the data sample; and

generating, for display within the graphical user interface of the client device, a modified interactive dataset visualization comprising a modified confusion matrix with a plurality of interactive cells mapped to the modified classifications.

18 . The method as recited in claim 17 , further comprising determining the one or more attributes associated with the natural language understanding task by determining a default computing hardware configuration or a user-defined computing hardware configuration for the natural language understanding task.

19 . The method as recited in claim 17 , wherein determining the cloud-based computing device comprises:

monitoring, by utilizing the scheduler, a status of plurality of cloud-based computing devices;

determining automatically, by utilizing the scheduler, that the cloud-based computing device comprises a status indicating availability of the plurality of cloud-based computing devices; and

determining automatically, by utilizing the scheduler, that the cloud-based computing device comprises computing hardware compatible with a computing hardware configuration indicated in the request to instantiate a selected machine-learning model.

20 . The method as recited in claim 17 , further comprising:

determining a plurality of annotated data samples from the dataset;

generating a plurality of intermediate representations of the plurality of annotated data samples, in response to detecting one or more modifications to the classifications mapped to the plurality of interactive cells; and

classifying, utilizing the selected machine-learning model, the plurality of annotated data samples based on the plurality of intermediate representations of the plurality of annotated data samples.

Assignments (1)

Continuity (1)

References Cited (39)