IP Library Granted Patent US 11,663,517
Granted Patent B2
US 11,663,517 · App. 15/884,878 · Granted May 30, 2023

Automatic machine learning model generation

Inventors: Sara Beth Asher (Atherton, CA); John Emery Ball (San Francisco, CA); Vitaly Gordon (Sunnyvale, CA); Till Christian Bergmann (San Mateo, CA); Kin Fai Kan (Sunnyvale, CA); Chalenge Masekera (Oakland, CA); Shubha Nabar (Sunnyvale, CA); Nihar Dandekar (Sunnyvale, CA); James Reber Lewis (San Francisco, CA)
Assignee: Salesforce, Inc.
G06N20/00G06N5/02G06N5/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,663,517
App. No.
15/884,878
Granted
May 30, 2023
Kind
B2
Abstract

A system may automatically generate a predictive machine learning model by automatically performing various processes based on an analysis of the data as well as metadata associated with the data. The system may accept a selection of data and a prediction field from the data. The system may automatically generate a set of features based on the data and may automatically remove certain features that cause inaccuracies in the model. The system may balance the data based on a representation rate of certain outcomes. The system may train and select a model based on several candidate models. The system may then perform the predictions based on the selected model and send an indication of the predictions to a user.

Claims (66)

1. A method for generating a predictive machine learning model at a database system, comprising:

receiving, from a cloud client of the database system via a user interface associated with the database system, a selection of a data set comprising a plurality of fields;

identifying metadata associated with the selected data set based at least in part on retrieving the selected data set from a data center of the database system, the metadata indicating respective data types and classifications of the plurality of fields in the selected data set;

receiving, from the cloud client via the user interface, a selection of a prediction field from the plurality of fields in the selected data set;

generating a plurality of features at a server of the database system based at least in part on the identified metadata associated with the selected data set, the plurality of features identifying relationships between the selected prediction field and other fields in the selected data set;

generating a plurality of candidate predictive machine learning models at the server based at least in part on the plurality of features;

selecting, from the plurality of candidate predictive machine learning models generated at the server, a predictive machine learning model that satisfies a threshold level of accuracy with respect to the selected prediction field; and

transmitting, to the cloud client for display in the user interface, an indication of a plurality of predicted values for the selected prediction field, wherein the plurality of predicted values are generated at the server using the selected predictive machine learning model.

2. The method of claim 1 , further comprising:

evaluating a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features; and

removing at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold.

3. The method of claim 2 , wherein the statistical relationship comprises a correlation calculation, a Cramer's V calculation, or a combination thereof.

4. The method of claim 1 , further comprising:

training the plurality of candidate predictive machine learning models using the plurality of features; and

evaluating the plurality of candidate predictive machine learning models based at least in part on a predictive accuracy of each of the plurality of candidate machine learning models wherein selecting the predictive machine learning model is based at least in part on the evaluating.

5. The method of claim 1 , further comprising:

adjusting a sampling rate of the selected data set based at least in part on a representation rate of the plurality of predicted values.

6. The method of claim 1 , further comprising:

determining a score for the prediction field based at least in part on generating the predictive machine learning model, wherein transmitting the indication of the plurality of predicted values is based at least in part on the determined score.

7. The method of claim 6 , further comprising:

receiving a new record associated with the selected data set; and

updating the score for the prediction field based at least in part on the new record.

8. The method of claim 6 , further comprising:

storing the score in a database server of the database system.

9. The method of claim 1 , wherein receiving the selection of the data set comprises:

receiving an indication of an object, wherein the indication of the plurality of predicted values is based at least in part on receiving the indication of the object.

10. The method of claim 9 , further comprising:

receiving an indication of one or more additional objects, wherein the indication of the plurality of predicted values is based at least in part on receiving the indication of the one or more additional objects.

11. The method of claim 1 , further comprising:

receiving a selection of prediction fields from the plurality of fields, wherein the plurality of features from the selected data set are generated based at least in part on the selection of prediction fields.

12. The method of claim 1 , further comprising:

saving the plurality of predicted values to the prediction field based at least in part on transmitting the indication of the plurality of predicted values.

13. The method of claim 1 , wherein the database system comprises a multi-tenant database system.

14. An apparatus for generating a predictive machine learning model at a database system, comprising:

a processor;

memory in electronic communication with the processor; and

instructions stored in the memory and executable by the processor to cause the apparatus to:

receive, from a cloud client of the database system via a user interface associated with the database system, a selection of a data set comprising a plurality of fields;

identify metadata associated with the selected data set based at least in part on retrieving the selected data set from a data center of the database system, the metadata indicating respective data types and classifications of the plurality of fields in the selected data set;

receive, from the cloud client via the user interface, a selection of a prediction field from the plurality of fields in the selected data set;

generate a plurality of features at a server of the database system based at least in part on the identified metadata associated with the selected data set, the plurality of features identifying relationships between the selected prediction field and other fields in the selected data set;

generate a plurality of candidate predictive machine learning models at the server based at least in part on the plurality of features;

select, from the plurality of candidate predictive machine learning models generated at the server, a predictive machine learning model that satisfies a threshold level of accuracy with respect to the selected prediction field; and

transmit, to the cloud client for display in the user interface, an indication of a plurality of predicted values for the selected prediction field, wherein the plurality of predicted values are generated at the server using the selected predictive machine learning model.

15. The apparatus of claim 14 , wherein the instructions are further executable by the processor to cause the apparatus to:

evaluate a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features; and

remove at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold.

16. The apparatus of claim 14 , wherein the instructions are further executable by the processor to cause the apparatus to:

train the plurality of candidate predictive machine learning models using the plurality of features; and

evaluate the plurality of candidate predictive machine learning models based at least in part on a predictive accuracy of each of the plurality of candidate machine learning models wherein selecting the predictive machine learning model is based at least in part on the evaluating.

17. The apparatus of claim 14 , wherein the instructions are further executable by the processor to cause the apparatus to:

adjust a sampling rate of the selected data set based at least in part on a representation rate of the plurality of predicted values.

18. A non-transitory computer-readable medium storing code for generating a predictive machine learning model at a database system, the code comprising instructions that are executable by a processor to:

receive, from a cloud client of the database system via a user interface associated with the database system, a selection of a data set comprising a plurality of fields;

identify metadata associated with the selected data set based at least in part on retrieving the selected data set from a data center of the database system, the metadata indicating respective data types and classifications of the plurality of fields in the selected data set;

receive, from the cloud client via the user interface, a selection of a prediction field from the plurality of fields in the selected data set;

generate a plurality of features at a server of the database system based at least in part on the identified metadata associated with the selected data set, the plurality of features identifying relationships between the selected prediction field and other fields in the selected data set;

generate a plurality of candidate predictive machine learning models at the server based at least in part on the plurality of features;

select, from the plurality of candidate predictive machine learning models generated at the server, a predictive machine learning model that satisfies a threshold level of accuracy with respect to the selected prediction field; and

transmit, to the cloud client for display in the user interface, an indication of a plurality of predicted values for the selected prediction field, wherein the plurality of predicted values are generated at the server using the selected predictive machine learning model.

19. The non-transitory computer-readable medium of claim 18 , wherein the instructions are further executable by the processor to:

evaluate a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features; and

remove at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold.

20. The method of claim 1 , wherein:

the selected data set comprises a table with a plurality of rows that include structured data; and

the prediction field comprises a row of the plurality of rows in the table.

Assignments (3)
CHANGE OF NAME Recorded Apr 19, 2023
From: SALESFORCE.COM, INC.
To: SALESFORCE, INC.
Reel/Frame 063398/0827 →
CORRECTIVE ASSIGNMENT TO CORRECT THE FIFTH ASSIGNOR PREVIOUSLY RECORDED AT REEL: 044814 FRAME: 0072. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Mar 7, 2018
From: ASHER, SARA BETH; BALL, JOHN EMERY; GORDON, VITALY; BERGMANN, TILL CHRISTIAN; KAN, KIN FAI; MASEKERA, CHALENGE; NABAR, SHUBHA; DANDEKAR, NIHAR; LEWIS, JAMES REBER
To: SALESFORCE.COM, INC.
Reel/Frame 045514/0178 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 2, 2018
From: ASHER, SARA BETH; BALL, JOHN EMERY; GORDON, VITALY; BERGMANN, TILL CHRISTIAN; FAI, KIN; MASEKERA, CHALENGE; NABAR, SHUBHA; DANDEKAR, NIHAR; LEWIS, JAMES REBER
To: SALESFORCE.COM, INC.
Reel/Frame 044814/0072 →
Continuity (2)
Provisional Application 62581624 · Nov 3, 2017
Related Publication 20190138946A1 · May 9, 2019
Cited By (1)
US 12,505,252