IP Library Granted Patent US 11,250,339
Granted Patent B2
US 11,250,339 · App. 15/387,302 · Granted Feb 15, 2022

Ensemble classification algorithms having subclass resolution

Inventors: Jonathan Sullivan (Hurricane, UT); Evan Brydon (New York, NY)
Assignee: The Nielsen Company (US), LLC
G06N7/005G06N20/00G06N20/20G06Q30/0202
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,250,339
App. No.
15/387,302
Granted
Feb 15, 2022
Kind
B2
Abstract

Ensemble classification algorithms having subclass resolution are disclosed. An example disclosed apparatus includes a fingerprint generator to generate a fingerprint of class probabilities of each of a plurality of samples, a distribution creator to create a distribution of the samples based on the generated fingerprints, and a distribution applicator to apply the distribution to a population to predict sub-class probabilities of each of the population.

Claims (69)

1. An apparatus comprising:

a fingerprint generator, implemented by at least one processor, to generate a fingerprint of primary class probabilities of respective ones of a plurality of samples retrieved from a database via a network, the fingerprint being a numerical value that is calculated based on class numbers that are ranked according to respective primary class probabilities;

a distribution creator, implemented by the at least one processor, to:

remove ones of the plurality of samples in which a respective predicted primary class prediction is different from an actual primary class, and

create a distribution of predicted subclass probabilities of the plurality of samples based on the generated fingerprints; and

a distribution applicator, implemented by the at least one processor, to:

apply the distribution of the predicted subclass probabilities to a population to predict sub-class probabilities of respective ones of the population,

determine a relevancy of ones of the predicted sub-class probabilities corresponding to the respective ones of the population in response to applying the distribution of the predicted subclass probabilities to the population, and

apply the predicted sub-class probabilities to the respective ones of the population based on the relevancy.

2. The apparatus as defined in claim 1 , wherein the distribution applicator is to assign ones of the population a sub-class based on the predicted sub-class probabilities.

3. The apparatus as defined in claim 1 , wherein the distribution creator is to pivot an array based on at least one fingerprint value of the distribution.

4. The apparatus as defined in claim 1 , wherein the distribution creator is to weight ones of the plurality of samples to create the distribution.

5. A method comprising:

generating, by executing an instruction with a processor, at least one fingerprint of primary class probabilities of respective ones of a plurality of samples retrieved from a database via a network, the fingerprint being a numerical value that is calculated based on class numbers that are ranked according to respective primary class probabilities;

comparing, by executing an instruction with the processor, a true primary class with a predicted primary class for the plurality of samples;

removing, by executing an instruction with the processor, respective ones of the plurality of samples in which a respective primary class does not match a predicted primary class;

creating, by executing an instruction with the processor, a distribution of subclass probabilities of the plurality of samples based on the at least one fingerprint;

applying, by executing an instruction with the processor, the distribution of the subclass probabilities to predict sub-class probabilities of respective ones of a population;

determining, by executing an instruction with the processor, a relevancy of ones of the predicted sub-class probabilities corresponding to the respective ones of the population in response to applying the distribution of the subclass probabilities to the population; and

determining, by executing an instruction with the processor, whether to apply the predicted sub-class probabilities to the respective ones of the population based on the relevancy.

6. The method as defined in claim 5 , wherein the creating of the distribution includes pivoting a table based on at least one fingerprint value.

7. The method as defined in claim 5 , wherein the generating of the at least one fingerprint includes arranging the plurality of samples in an array.

8. The method as defined in claim 7 , further including pivoting, by executing an instruction with the processor, the array based on attributes.

9. The method as defined in claim 7 , further including removing, by executing an instruction with the processor, samples of the array in which a predicted primary class does not match an actual primary class.

10. The method as defined in claim 7 , further including assigning, by executing an instruction with the processor, at least a portion of the population with a sub-class based on a respective class probability.

11. The method as defined in claim 7 , wherein the creating of the distribution of the subclass probabilities of the plurality of samples includes weighting, by executing an instruction with the processor, at least one of the population.

12. A tangible machine readable medium comprising instructions, which when executed, cause a processor to at least:

generate at least one fingerprint of primary class probabilities for respective ones of a plurality of samples retrieved from a database via a network, the fingerprint being a numerical value that is calculated based on class numbers that are ranked according to respective primary class probabilities;

compare a true primary class with a predicted primary class for the ones of the samples;

remove ones of the samples in which a primary class does not match a respective predicted primary class;

create a distribution of predicted sub-class probabilities based on the at least one fingerprint; and

apply the distribution of the predicted subclass probabilities to predict sub-class probabilities of respective ones of a population;

determine a relevancy of ones of the predicted sub-class probabilities corresponding to the respective ones of the population in response to applying the distribution of the predicted subclass probabilities to the population; and

apply the predicted sub-class probabilities to the respective ones of the population based on the relevancy.

13. The machine readable medium as defined in claim 12 , wherein the at least one fingerprint is generated by arranging ones of the samples in an array.

14. The machine readable medium as defined in claim 13 , wherein the instructions cause the processor to pivot the array based on at least one fingerprint value of the at least one fingerprint.

15. The machine readable medium as defined in claim 12 , wherein the instructions cause the processor to assign at least one of the population with a sub-class based on a respective sub-class probability.

16. The machine readable medium as defined in claim 12 , wherein the instructions cause the processor to sort samples of the sample group.

17. The machine readable medium as defined in claim 12 , wherein the distribution is created based on weighted samples of the plurality of samples.

18. The apparatus as defined in claim 1 , wherein the distribution applicator is to determine the relevancy based on whether ones of the predicted subclass probabilities are possible in relation to the ones of the population.

19. The apparatus as defined in claim 1 , wherein the predicted sub-class probabilities are not applied to the ones of the population having a relevancy below a relevancy threshold.

20. The apparatus as defined in claim 1 , wherein the distribution of predicted subclass probabilities is not associated with a decision tree.

21. The apparatus as defined in claim 1 , wherein the distribution creator is to apply weighting factors to ones of the plurality of samples, the weighting factors based on a reliability of sources of the plurality of samples.

22. The apparatus as defined in claim 1 , wherein the distribution creator is to apply the predicted sub-class probabilities to the ones of the population determined to be relevant.

23. An apparatus comprising:

a fingerprint generator, implemented by at least one processor, to generate a fingerprint of primary class probabilities of respective ones of a plurality of samples retrieved from a database via a network, the fingerprint generated based on an equation, f=Σ i=1 c r i *n (c-i) , wherein r is a class number that is numbered based on a ranked order of respective probabilities, p, and wherein n is a total number of classes and c is a configurable parameter to allow for a bias variance trade-off;

a distribution creator, implemented by the at least one processor, to:

remove ones of the plurality of samples in which a respective predicted primary class prediction is different from an actual primary class, and

create a distribution of predicted subclass probabilities of the plurality of samples based on the generated fingerprints; and

a distribution applicator, implemented by the at least one processor, to:

apply the distribution of the predicted subclass probabilities to a population to predict sub-class probabilities of respective ones of the population,

determine a relevancy of ones of the predicted sub-class probabilities corresponding to the respective ones of the population in response to applying the distribution of the predicted subclass probabilities to the population, and

apply the predicted sub-class probabilities to the respective ones of the population based on the relevancy.

24. An apparatus, comprising:

at least one memory;

instructions in the apparatus; and

processor circuitry to execute the instructions to:

generate at least one fingerprint of primary class probabilities for respective ones of a plurality of samples retrieved from a database via a network, the fingerprint being a numerical value that is calculated based on class numbers that are ranked according to respective primary class probabilities;

compare a true primary class with a predicted primary class for the ones of the samples;

remove ones of the samples in which a primary class does not match a respective predicted primary class;

create a distribution of predicted sub-class probabilities based on the at least one fingerprint; and

apply the distribution of the predicted subclass probabilities to predict sub-class probabilities of respective ones of a population;

determine a relevancy of ones of the predicted sub-class probabilities corresponding to the respective ones of the population in response to applying the distribution of the predicted subclass probabilities to the population; and

apply the predicted sub-class probabilities to the respective ones of the population based on the relevancy.

25. The apparatus as defined in claim 24 , wherein the at least one fingerprint is generated by arranging ones of the samples in an array.

26. The apparatus as defined in claim 25 , wherein the processor circuitry is to execute the instructions to pivot the array based on at least one fingerprint value of the at least one fingerprint.

27. The apparatus as defined in claim 24 , wherein the processor circuitry is to execute the instructions to assign at least one of the population with a sub-class based on a respective sub-class probability.

28. The apparatus as defined in claim 24 , wherein the processor circuitry is to execute the instructions to sort samples of the sample group.

29. The apparatus as defined in claim 24 , wherein the distribution is created based on weighted samples of the plurality of samples.

Assignments (8)
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
SECURITY INTEREST Recorded May 8, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: ARES CAPITAL CORPORATION
Reel/Frame 063574/0632 →
SECURITY INTEREST Recorded Apr 28, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: CITIBANK, N.A.
Reel/Frame 063561/0381 →
SECURITY AGREEMENT Recorded Jan 31, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: BANK OF AMERICA, N.A.
Reel/Frame 063560/0547 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 14, 2017
From: SULLIVAN, JONATHAN; BRYDON, EVAN
To: THE NIELSEN COMPANY (US), LLC
Reel/Frame 041570/0753 →
Continuity (2)
Provisional Application 62353341 · Jun 22, 2016
Related Publication 20170372213A1 · Dec 28, 2017