IP Library Granted Patent US 11,080,616
Granted Patent B2
US 11,080,616 · App. 15/715,433 · Granted Aug 3, 2021

Artificial intelligence model and data collection/development platform

Inventors: Matthew Zeiler (New York, NY); Daniel Kantor (New York, NY); Christopher Fox (New York, NY); Cassidy Williams (New York, NY)
Assignee: CLARIFAI, INC.
G06N20/00G06F3/048G06F8/00G06F8/31G06F8/65G06N3/0454G06N5/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,080,616
App. No.
15/715,433
Filed
Sep 26, 2017
Granted
Aug 3, 2021
Kind
B2
Art Unit
2125
USPC
706/12
Abstract

In some embodiments, a service platform that facilitates artificial intelligence model and data collection and collection may be provided. Input/output information derived from machine learning models may be obtained via the service platform. The input/output information may indicate (i) first items provided as input to at least one model of the machine learning models, (ii) first prediction outputs derived from the at least one model's processing of the first items, (iii) second items provided as input to at least another model of the machine learning models, (iv) second prediction outputs derived from the at least one other model's processing of the second items, and (v) other inputs and outputs. The input/output information may be provided via the service platform to update a first machine learning model. The first machine learning model may be updated based on the input/output information being provided as input to the first machine learning model.

Claims (68)

1. A method of facilitating development of a machine learning model via a service platform, the method being implemented by a computer system that comprises one or more processors executing computer program instructions that, when executed, perform the method, the method comprising:

providing a service platform that enables a developer to obtain training item information for training a machine learning model, the training item information comprising inputs and prediction outputs derived from one or more machine learning models' processing of the inputs;

obtaining, via the service platform, a request for information to train a first machine learning model, the request indicating a requested concept on which the first machine learning model is to be trained;

obtaining, via the service platform, input and output information derived from machine learning models, the input and output information comprising (i) first items provided as input to at least one model of the machine learning models, (ii) first prediction outputs derived from the at least one model's processing of the first items, (iii) second items provided as input to at least one other model of the machine learning models, and (iv) second prediction outputs derived from the at least one other model's processing of the second items; and

providing, via the service platform, based on the requested concept, at least a subset of the input and output information derived from the machine learning models to update the first machine learning model, such that the subset of the input and output information comprises:

(i) the first items and the first prediction outputs in response to the first prediction outputs being determined to correspond to a first concept that matches the requested concept; and

(ii) the second items and the second prediction outputs in response to the second prediction outputs being determined to correspond to a second concept that matches the requested concept, the second concept being different from the first concept,

wherein the first machine learning model is updated based on the subset of the input and output information being provided as input to the first machine learning model.

2. The method of claim 1 , wherein the first machine learning model is stored as a first instance at the service platform, the method further comprising:

generating a second instance of the first machine learning model,

wherein providing the subset of the input and output information to update the first machine learning model comprises providing the subset of the input and output information as input to the second instance of the first machine learning model to train the second instance of the first machine learning model, the training of the first machine learning model based on the subset of the input and output information causing updating of the second instance of the first machine learning model;

analyzing accuracy of the updated second instance of the first machine learning model; and

causing, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

3. The method of claim 2 , wherein one or more machine learning models hosted at the service platform (i) analyzes the accuracy of the updated second instance of the first machine learning model and (ii) causes, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

4. The method of claim 2 , wherein one or more machine learning models hosted at the service platform (i) provides the subset of the input and output information as input to the second instance of the first machine learning model to train the second instance of the first machine learning model, (ii) analyzes the accuracy of the updated second instance of the first machine learning model, and (iii) causes, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

5. The method of claim 2 , wherein one or more machine learning models hosted at the service platform (i) generates the second instance of the first machine learning model, (ii) provides the subset of the input and output information as input to the second instance of the first machine learning model to train the second instance of the first machine learning model, (iii) analyzes the accuracy of the updated second instance of the first machine learning model, and (iv) causes, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

6. The method of claim 1 , wherein the first machine learning model is stored as a first instance at the service platform, the method further comprising:

generating a second instance of the first machine learning model;

obtaining code information indicating one or more code-related changes for the first machine learning model;

updating the second instance of the first machine learning model based on the code information;

analyzing accuracy, speed, or cost of the updated second instance of the first machine learning model; and

causing, based on the accuracy or cost, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

7. The method of claim 6 , wherein one or more machine learning models hosted at the service platform (i) generates the second instance of the first machine learning model, (ii) updates the second instance of the first machine learning model based on the code information, (iii) analyzes the accuracy or cost of the updated second instance of the first machine learning model, and (iv) causes, based on the accuracy or cost, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

8. The method of claim 1 , wherein the request indicates a domain associated with the first machine learning model, and wherein providing the subset of the input and output information comprises:

determining a first domain associated with at least one machine learning model from which the first prediction outputs are derived;

determining that the first domain matches the requested domain; and

in response to the first domain being determined to match the requested domain and the first prediction outputs being determined to correspond to the first concept matching the requested concept, including the first items and the first prediction outputs in the subset of the input and output information to be provided to update the first machine learning machine.

9. The method of claim 1 , wherein the request indicates an input type associated with the first machine learning model, and wherein providing the subset of the input and output information comprises:

determining a first input type associated with at least one machine learning model from which the first prediction outputs are derived;

determining that the first input type matches the requested input type; and

in response to the first input type being determined to match the requested input type and the first prediction outputs being determined to correspond to the first concept matching the requested concept, including the first items and the first prediction outputs in the subset of the input and output information to be provided to update the first machine learning machine.

10. A system comprising:

a computer system that comprises one or more processors programmed with computer program instructions that, when executed, cause the computer system to:

provide a service platform that enables a developer to obtain training item information for training a machine learning model, the training item information comprising inputs and prediction outputs derived from one or more machine learning models' processing of the inputs;

obtain, via the service platform, a request for information to train a first machine learning model, the request indicating a requested concept on which the first machine learning model is to be trained;

obtain, via the service platform, input and output information derived from machine learning models, the input and output information comprising (i) first items provided as input to at least one model of the machine learning models, (ii) first prediction outputs derived from the at least one model's processing of the first items, (iii) second items provided as input to at least one other model of the machine learning models, and (iv) second prediction outputs derived from the at least one other model's processing of the second items; and

provide, via the service platform, based on the requested concept, at least a subset of the input and output information derived from the machine learning models to update the first machine learning model, such that the subset of the input and output information comprises:

(i) the first items and the first prediction outputs in response to the first prediction outputs being determined to correspond to a first concept that matches the requested concept; and

(ii) the second items and the second prediction outputs in response to the second prediction outputs being determined to correspond to a second concept that matches the requested concept, the second concept being different from the first concept,

wherein the first machine learning model is updated based on the subset of the input and output information being provided as input to the first machine learning model.

11. The system of claim 10 , wherein the first machine learning model is stored as a first instance at the service platform, and wherein the computer system is caused to:

generate a second instance of the first machine learning model,

wherein providing the subset of the input and output information to update the first machine learning model comprises providing the subset of the input and output information as input to the second instance of the first machine learning model to train the second instance of the first machine learning model, the training of the first machine learning model based on the subset of the input and output information causing updating of the second instance of the first machine learning model;

analyze accuracy of the updated second instance of the first machine learning model; and

cause, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

12. The system of claim 11 , wherein one or more machine learning models hosted at the service platform (i) analyzes the accuracy of the updated second instance of the first machine learning model and (ii) causes, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

13. The system of claim 11 , wherein one or more machine learning models hosted at the service platform (i) provides the subset of the input and output information as input to the second instance of the first machine learning model to train the second instance of the first machine learning model, (ii) analyzes the accuracy of the updated second instance of the first machine learning model, and (iii) causes, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

14. The system of claim 11 , wherein one or more machine learning models hosted at the service platform (i) generates the second instance of the first machine learning model, (ii) provides the subset of the input and output information as input to the second instance of the first machine learning model to train the second instance of the first machine learning model, (iii) analyzes the accuracy of the updated second instance of the first machine learning model, and (iv) causes, based on the accuracy, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

15. The system of claim 10 , wherein the first machine learning model is stored as a first instance at the service platform, and wherein the computer system is caused to:

generate a second instance of the first machine learning model;

obtain code information indicating one or more code-related changes for the first machine learning model;

update the second instance of the first machine learning model based on the code information;

analyze accuracy or cost of the updated second instance of the first machine learning model; and

cause, based on the accuracy or cost, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

16. The system of claim 15 , wherein one or more machine learning models hosted at the service platform (i) generates the second instance of the first machine learning model, (ii) updates the second instance of the first machine learning model based on the code information, (iii) analyzes the accuracy or cost of the updated second instance of the first machine learning model, and (iv) causes, based on the accuracy or cost, the updated second instance of the first machine learning model to be stored at the service platform as a version of the first machine learning model.

17. The system of claim 10 , wherein the request indicates a domain associated with the first machine learning model, and wherein providing the subset of the input and output information comprises:

determining a first domain associated with at least one machine learning model from which the first prediction outputs are derived;

determining that the first domain matches the requested domain; and

in response to the first domain being determined to match the requested domain and the first prediction outputs being determined to correspond to the first concept matching the requested concept, including the first items and the first prediction outputs in the subset of the input and output information to be provided to update the first machine learning machine.

18. The system of claim 10 , wherein the request indicates an input type associated with the first machine learning model, and wherein providing the subset of the input and output information comprises:

determining a first input type associated with at least one machine learning model from which the first prediction outputs are derived;

determining that the first input type matches the requested input type; and

in response to the first input type being determined to match the requested input type and the first prediction outputs being determined to correspond to the first concept matching the requested concept, including the first items and the first prediction outputs in the subset of the input and output information to be provided to update the first machine learning machine.

19. One or more non-transitory storage media comprising instructions that, when executed by one or more processors, cause operations comprising: providing a service platform that enables a developer to obtain training item information for training a machine learning model, the training item information comprising inputs and prediction outputs derived from one or more machine learning models' processing of the inputs; obtaining, via the service platform, a request for information to train a first machine learning model, the request indicating a requested concept on which the first machine learning model is to be trained; obtaining, via the service platform, input and output information derived from machine learning models, the input and output information comprising (i) first items provided as input to at least one model of the machine learning models, (ii) first prediction outputs derived from the at least one model's processing of the first items, (iii) second items provided as input to at least one other model of the machine learning models, and (iv) second prediction outputs derived from the at least one other model's processing of the second items; and providing, via the service platform, based on the requested concept, at least a subset of the input and output information derived from the machine learning models to update the first machine learning model, such that the subset of the input and output information comprises: (i) the first items and the first prediction outputs in response to the first prediction outputs being determined to correspond to a first concept that matches the requested concept; and (ii) the second items and the second prediction outputs in response to the second prediction outputs being determined to correspond to a second concept that matches the requested concept, the second concept being different from the first concept, wherein the first machine learning model is updated based on the subset of the input and output information being provided as input to the first machine learning model.

20. The media of claim 19 , wherein the request indicates a domain associated with the first machine learning model, and wherein providing the subset of the input and output information comprises:

determining a first domain associated with at least one machine learning model from which the first prediction outputs are derived;

determining that the first domain matches the requested domain; and

in response to the first domain being determined to match the requested domain and the first prediction outputs being determined to correspond to the first concept matching the requested concept, including the first items and the first prediction outputs in the subset of the input and output information to be provided to update the first machine learning machine.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 11, 2026
From: CLARIFAI, INC.
To: NEBIUS BV
Reel/Frame 075712/0109 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2018
From: ZEILER, MATTHEW; FOX, CHRISTOPHER; WILLIAMS, CASSIDY
To: CLARIFAI, INC.
Reel/Frame 047811/0617 →
EMPLOYEE AGREEMENT Recorded Dec 18, 2018
From: KANTOR, DANIEL
To: CLARIFAI, INC.
Reel/Frame 047952/0456 →
Continuity (2)
Provisional Application 62400543 · Sep 27, 2016
Related Publication 20180089591A1 · Mar 29, 2018
Cited By (1)
US 12,505,097