IP Library Granted Patent US 11,615,366
Granted Patent B2
US 11,615,366 · App. 16/849,199 · Granted Mar 28, 2023

Evaluation of product-related data structures using machine-learning techniques

Inventors: Amihai Savir (Sansana, IL); Arthur Wensing (Nieuw-Vennep, NL); Noga Gershon (Dimona, IL); Marcel Bernard Körner (Chorvatsky Grob, SK); Ivan Mlynek (D. Jurkovica, SK); Jorge Luis Perez (Coral Gables, FL); Michael Rupert James Thatcher (Streatley, GB); Dhev George Kollannur (Austin, TX); Omer Sagi (Mazkaret Batya, IL)
Assignee: EMC IP Holding Company LLC
G06Q10/06395G06N5/04G06N20/00G06Q10/087G06Q10/10G06Q30/0185
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,615,366
App. No.
16/849,199
Granted
Mar 28, 2023
Kind
B2
Abstract

Artificial intelligence (AI)-based techniques are provided that predict a quality score for a product-related data structure associated with one or more products. One method comprises obtaining data for a given product-related data structure; evaluating a plurality of first features related to a customer account associated with the given product-related data structure using the obtained data; evaluating a plurality of second features related to the given product-related data structure using the obtained data; processing at least some of the first features and the second features using at least one model that provides a predicted quality score for the given product-related data structure; and applying one or more thresholds to the predicted quality score to determine an acceptance status related to the given product-related data structure. A weighting of the first features and the second features can be learned during a training phase.

Claims (49)

1. A method, comprising:

obtaining data for a given product-related data structure;

evaluating a plurality of first features related to a customer account associated with the given product-related data structure using the obtained data;

evaluating a plurality of second features using the obtained data from the given product-related data structure;

training at least one machine learning model during a training phase by evaluating the plurality of first features and the plurality of second features using historical training data comprising at least one acceptance status label such that the at least one machine learning model learns (i) to predict a predicted quality score and (ii) a weighting of one or more first features and one or more second features, wherein the weighting is based at least in part on a feature importance of the one or more first features and a feature importance of the one or more second features, wherein the weighting comprises a first weighting of the one or more first features used to calculate a first score and a second weighting of the one or more second features used to calculate a second score;

implementing the at least one machine learning model using at least one processing device comprising a processor coupled to a memory;

applying the one or more first features and the one or more second features to the at least one machine learning model that predicts the predicted quality score for the given product-related data structures;

predicting, using the at least one machine learning model, the predicted quality score for the given product-related data structure, wherein the predicted quality score for the given product-related data structure comprises an aggregation based at least in part on the first score and the second score;

applying, using the at least one processing device, one or more thresholds to the predicted quality score to automatically determine an acceptance status related to the given product-related data structure; and

automatically initiating a processing of the given product-related data structure based at least in part on one or more of the acceptance status and the predicted quality score, wherein the automatically initiating the processing of the given product-related data structure comprises one or more of: (i) initiating a generation of an automated acceptance related to the given product-related data structure based at least in part on the acceptance status; (ii) initiating a generation of an automated denial related to the given product-related data structure based at least in part on the acceptance status; and (iii) initiating a prioritization of the given product-related data structure for a manual review based at least in part on the predicted quality score.

2. The method of claim 1 , wherein the acceptance status comprises one or more of an automatically accepted status, an automatically denied status and an additional review required status.

3. The method of claim 2 , wherein the acceptance status of the given product-related data structure comprises the automatically accepted status in response to the predicted quality score for the given product-related data structure exceeding a corresponding acceptance threshold, and wherein the acceptance status of the given product-related data structure comprises the automatically denied status in response to the predicted quality score for the given product-related data structure being below a corresponding denial threshold.

4. The method of claim 3 , wherein the acceptance status of the given product-related data structure comprises the additional review required status in response to the predicted quality score for the given product-related data structure being between the corresponding acceptance threshold and the corresponding denial threshold.

5. The method of claim 1 , further comprising at least one statistical model that determines a second predicted quality score for the given product-related data structure and further comprising selecting one of (i) the predicted quality score for the given product-related data structure predicted by the at least one machine learning model and (ii) the second predicted quality score for the given product-related data structure determined by the at least one statistical model.

6. The method of claim 1 , wherein the at least one machine learning model compares the given product-related data structure to similar product-related data structures based on one or more similarity criteria to evaluate a quality of the given product-related data structure.

7. The method of claim 1 , wherein the aggregation comprises an aggregation of at least two of an account score, a product-related data structure score and a product score for at least one product associated with the given product-related data structure.

8. The method of claim 1 , wherein the given product-related data structure comprises a plurality of items, and wherein a predicted quality score is separately determined for each of the plurality of items and wherein the aggregation further comprises a weighted aggregation of the predicted quality score for each of the plurality of items.

9. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

the at least one processing device being configured to implement the following steps:

obtaining data for a given product-related data structure;

evaluating a plurality of first features related to a customer account associated with the given product-related data structure using the obtained data;

evaluating a plurality of second features using the obtained data from the given product-related data structure;

training at least one machine learning model during a training phase by evaluating the plurality of first features and the plurality of second features using historical training data comprising at least one acceptance status label such that the at least one machine learning model learns (i) to predict a predicted quality score and (ii) a weighting of one or more first features and one or more second features, wherein the weighting is based at least in part on a feature importance of the one or more first features and a feature importance of the one or more second features, wherein the weighting comprises a first weighting of the one or more first features used to calculate a first score and a second weighting of the one or more second features used to calculate a second score;

implementing the at least one machine learning model using the at least one processing device;

applying the one or more first features and the one or more second features to the at least one machine learning model that predicts the predicted quality score for the given product-related data structure;

predicting, using the at least one machine learning model, the predicted quality score for the given product-related data structure, wherein the predicted quality score for the given product-related data structure comprises an aggregation based at least in part on the first score and the second score;

applying, using the at least one processing device, one or more thresholds to the predicted quality score to automatically determine an acceptance status related to the given product-related data structure; and

automatically initiating a processing of the given product-related data structure based at least in part on one or more of the acceptance status and the predicted quality score, wherein the automatically initiating the processing of the given product-related data structure comprises one or more of: (i) initiating a generation of an automated acceptance related to the given product-related data structure based at least in part on the acceptance status; (ii) initiating a generation of an automated denial related to the given product-related data structure based at least in part on the acceptance status; and (iii) initiating a prioritization of the given product-related data structure for a manual review based at least in part on the predicted quality score.

10. The apparatus of claim 9 , wherein the at least one machine learning model compares the given product-related data structure to similar product-related data structures based on one or more similarity criteria to evaluate a quality of the given product-related data structure.

11. The apparatus of claim 9 , wherein the aggregation comprises an aggregation of at least two of an account score, a product-related data structure score and a product score for at least one product associated with the given product-related data structure.

12. The apparatus of claim 9 , wherein the acceptance status comprises one or more of an automatically accepted status, an automatically denied status and an additional review required status.

13. The apparatus of claim 12 , wherein the acceptance status of the given product-related data structure comprises the automatically accepted status in response to the predicted quality score for the given product-related data structure exceeding a corresponding acceptance threshold, and wherein the acceptance status of the given product-related data structure comprises the automatically denied status in response to the predicted quality score for the given product-related data structure being below a corresponding denial threshold.

14. The apparatus of claim 13 , wherein the acceptance status of the given product-related data structure comprises the additional review required status in response to the predicted quality score for the given product-related data structure being between the corresponding acceptance threshold and the corresponding denial threshold.

15. The apparatus of claim 9 , wherein the given product-related data structure comprises a plurality of items, and wherein a predicted quality score is separately determined for each of the plurality of items and wherein the aggregation further comprises a weighted aggregation of the predicted quality score for each of the plurality of items.

16. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:

obtaining data for a given product-related data structure;

evaluating a plurality of first features related to a customer account associated with the given product-related data structure using the obtained data;

evaluating a plurality of second features using of the obtained data from the given product-related data structure;

training at least one machine learning model during a training phase by evaluating the plurality of first features and the plurality of second features using historical training data comprising at least one acceptance status label such that the at least one machine learning model learns (i) to predict a predicted quality score and (ii) a weighting of one or more first features and one or more second features, wherein the weighting is based at least in part on a feature importance of the one or more first features and a feature importance of the one or more second features, wherein the weighting comprises a first weighting of the one or more first features used to calculate a first score and a second weighting of the one or more second features used to calculate a second score;

implementing the at least one machine learning model using the at least one processing device;

applying the one or more first features and the one or more second features to the at least one machine learning model that predicts the predicted quality score for the given product-related data structure;

predicting, using the at least one machine learning model, the predicted quality score for the given product-related data structure, wherein the predicted quality score for the given product-related data structure comprises an aggregation based at least in part on the first score and the second score;

applying, using the at least one processing device, one or more thresholds to the predicted quality score to automatically determine an acceptance status related to the given product-related data structure; and

automatically initiating a processing of the given product-related data structure based at least in part on one or more of the acceptance status and the predicted quality score, wherein the automatically initiating the processing of the given product-related data structure comprises one or more of: (i) initiating a generation of an automated acceptance related to the given product-related data structure based at least in part on the acceptance status; (ii) initiating a generation of an automated denial related to the given product-related data structure based at least in part on the acceptance status; and (iii) initiating a prioritization of the given product-related data structure for a manual review based at least in part on the predicted quality score.

17. The non-transitory processor-readable storage medium of claim 16 , wherein the acceptance status of the given product-related data structure comprises one or more of an automatically accepted status in response to the predicted quality score for the given product-related data structure exceeding a corresponding acceptance threshold, an automatically denied status in response to the predicted quality score for the given product-related data structure being below a corresponding denial threshold, and an additional review required status in response to the predicted quality score for the given product-related data structure being between the corresponding acceptance threshold and the corresponding denial threshold.

18. The non-transitory processor-readable storage medium of claim 16 , wherein the at least one machine learning model compares the given product-related data structure to similar product-related data structures based on one or more similarity criteria to evaluate a quality of the given product-related data structure.

19. The non-transitory processor-readable storage medium of claim 16 , wherein the aggregation comprises an aggregation of at least two of an account score, a product-related data structure score and a product score for at least one product associated with the given product-related data structure.

20. The non-transitory processor-readable storage medium of claim 16 , wherein the given product-related data structure comprises a plurality of items, and wherein a predicted quality score is separately determined for each of the plurality of items and wherein the aggregation further comprises a weighted aggregation of the predicted quality score for each of the plurality of items.

Assignments (9)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (052851/0081) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC
Reel/Frame 060436/0441 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (052851/0917) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC
Reel/Frame 060436/0509 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (052852/0022) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC
Reel/Frame 060436/0582 →
RELEASE OF SECURITY INTEREST AT REEL 052771 FRAME 0906 Recorded Nov 2, 2021
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
To: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC
Reel/Frame 058001/0298 →
SECURITY INTEREST Recorded Jun 5, 2020
From: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT
Reel/Frame 052852/0022 →
SECURITY INTEREST Recorded Jun 5, 2020
From: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC; THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT
Reel/Frame 052851/0081 →
SECURITY INTEREST Recorded Jun 5, 2020
From: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT
Reel/Frame 052851/0917 →
SECURITY AGREEMENT Recorded May 28, 2020
From: DELL PRODUCTS L.P.; EMC IP HOLDING COMPANY LLC
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
Reel/Frame 052771/0906 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 15, 2020
From: SAVIR, AMIHAI; WENSING, ARTHUR; GERSHON, NOGA; KORNER, MARCEL BERNARD; MLYNEK, IVAN; PEREZ, JORGE LUIS; THATCHER, MICHAEL RUPERT JAMES; KOLLANNUR, DHEV GEORGE; SAGI, OMER
To: EMC IP HOLDING COMPANY LLC
Reel/Frame 052404/0031 →