IP Library › Granted Patent US 12,387,733
Granted Patent B2
US 12,387,733 · App. 16/453,654 · Granted Aug 12, 2025

Methods and apparatus to fingerprint an audio signal via normalization

Inventors: Robert Coover (Orinda, CA); Zafar Rafii (Berkeley, CA)
Assignee: GRACENOTE, INC.
G10L19/02G10L25/18G10L25/51
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,387,733
App. No.
16/453,654
Granted
Aug 12, 2025
Kind
B2
Abstract

Methods, apparatus, systems, and articles of manufacture are disclosed to fingerprint audio via mean normalization. An example apparatus for audio fingerprinting includes a frequency range separator to transform an audio signal into a frequency domain, the transformed audio signal including a plurality of time-frequency bins including a first time-frequency bin, an audio characteristic determiner to determine a first characteristic of a first group of time-frequency bins of the plurality of time-frequency bins, the first group of time-frequency bins surrounding the first time-frequency bin and a signal normalizer to normalize the audio signal to thereby generate normalized energy values, the normalizing of the audio signal including normalizing the first time-frequency bin by the first characteristic. The example apparatus further includes a point selector to select one of the normalized energy values and a fingerprint generator to generate a fingerprint of the audio signal using the selected one of the normalized energy values.

Claims (79)

1. An apparatus for audio fingerprinting, comprising:

a frequency range separator to transform an audio signal into a frequency domain, the transformed audio signal including a plurality of time-frequency bins, each of the time-frequency bins corresponding to an intersection of a frequency bin and a time bin and contains a portion of the audio signal;

an audio characteristic determiner to:

select a first time-frequency bin;

determine a group of the plurality of time-frequency bins based on the first time-frequency bin and time-frequency bins within a pre-defined distance of the first time-frequency bin;

determine an audio characteristic for an audio region comprising the group of the plurality of time-frequency bins, wherein the determined audio characteristic for the audio region includes at least one of: (i) a mean energy value; (ii) a mode energy value (iii) an average power value; (iv) a mode power value; or (v) a mean amplitude of the group of the plurality of time-frequency bins;

select a second time-frequency bin;

determine a second group of the plurality of time-frequency bins based on the second time-frequency bin and time-frequency bins within a pre-defined distance of the second time-frequency bin, wherein at least a portion of the group of time-frequency bins overlaps at least a portion of the second group of time-frequency bins; and

determine a second audio characteristic for a second audio region comprising the second group of the plurality of time frequency bins;

a signal normalizer to:

normalize the audio region to generate normalized energy values, wherein normalizing the audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the group of the plurality of time-frequency bins based on the determined audio characteristic associated with the audio region; and

normalize the second audio region to generate second normalized energy values, wherein normalizing the second audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the second group of the plurality of time-frequency bins based on the determined second audio characteristic associated with the second audio region;

a point selector configured to:

determine a category of the audio signal;

weigh each of the time-frequency bins of the group of the plurality of time-frequency bins based on the determined category of the audio signal; and

weigh the selecting of the one of the normalized energy values by the category of the audio signal; and

select one of the normalized energy values; and

a fingerprint generator to generate a fingerprint of the audio signal using the selected one of the normalized energy values.

2. The apparatus of claim 1 , wherein the frequency range separator is further configured to perform a fast Fourier transform of the audio signal.

3. The apparatus of claim 1 , wherein the category of the audio signal includes at least one or music, human speech, sound effects, or advertisement.

4. The apparatus of claim 1 , wherein the point selector selects the one of the normalized energy values based on an energy extrema of the normalized audio region.

5. The apparatus of claim 1 , wherein each time-frequency bin of the plurality of time-frequency bins is a unique combination of (1) a time period of the transformed audio signal and (2) a frequency bin of the transformed audio signal.

6. A method for audio fingerprinting, comprising:

transforming an audio signal into a frequency domain, the transformed audio signal including a plurality of time-frequency bins, each of the time-frequency bins corresponding to an intersection of a frequency bin and a time bin and contains a portion of the audio signal;

selecting a first time-frequency bin;

determining a group of the plurality of time-frequency bins based on the first time-frequency bin and time-frequency bins within a pre-defined distance of the first time-frequency bin;

determining an audio characteristic for an audio region comprising the group of the plurality of time-frequency bins, wherein the determined audio characteristic for the audio region includes at least one of: (i) a mean energy value; (ii) a mode energy value (iii) an average power value; (iv) a mode power value; or (v) a mean amplitude of the group of the plurality of time-frequency bins;

selecting a second time-frequency bin;

determining a second group of the plurality of time-frequency bins based on the second time-frequency bin and time-frequency bins within a pre-defined distance of the second time-frequency bin, wherein at least a portion of the group of time-frequency bins overlaps at least a portion of the second group of time-frequency bins;

determining a second audio characteristic for a second audio region comprising the second group of the plurality of time frequency bins;

normalizing the audio region to generate normalized energy values, wherein normalizing the audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the group of the plurality of time-frequency bins based on the determined audio characteristic associated with the audio region;

normalizing the second audio region to generate second normalized energy values, wherein normalizing the second audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the second group of the plurality of time-frequency bins based on the determined second audio characteristic associated with the second audio region;

selecting one of the normalized energy values, wherein selecting one of the normalized energy values comprises:

determining a category of the audio signal;

weighing each of the time-frequency bins of the group of the plurality of time-frequency bins based on the determined category of the audio signal; and

weighing the selecting of the one of the normalized energy values by the category of the audio signal; and

generating a fingerprint of the audio signal using the selected one of the normalized energy values.

7. The method of claim 6 , wherein the transforming the audio signal into the frequency domain includes performing a fast Fourier transform of the audio signal.

8. The method of claim 6 , wherein the category of the audio signal includes at least one of music, human speech, sound effects, or advertisement.

9. The method of claim 6 , wherein the selecting the one of the normalized energy values is based on an energy extrema of the normalized audio region.

10. The method of claim 6 , wherein each time-frequency bin of the plurality of time-frequency bins is a unique combination of (1) a time period of the transformed audio signal and (2) a frequency bin of the transformed audio signal.

11. A non-transitory computer readable storage medium comprising instructions which, when executed, cause a processor to at least:

transform an audio signal into a frequency domain, the transformed audio signal including a plurality of time-frequency bins, each of the time-frequency bins corresponding to an intersection of a frequency bin and a time bin and contains a portion of the audio signal;

select a first time-frequency bin;

determine a group of the plurality of time-frequency bins based on the first time-frequency bin and time-frequency bins within a pre-defined distance of the first time-frequency bin;

determine an audio characteristic for an audio region comprising the group of the plurality of time-frequency bins, wherein the determined audio characteristic for the audio region includes at least one of: (i) a mean energy value; (ii) a mode energy value (iii) an average power value; (iv) a mode power value; or (v) a mean amplitude of the group of the plurality of time-frequency bins;

select a second time-frequency bin;

determine a second group of the plurality of time-frequency bins based on the second time-frequency bin and time-frequency bins within a pre-defined distance of the second time-frequency bin, wherein at least a portion of the group of time-frequency bins overlaps at least a portion of the second group of time-frequency bins;

determine a second audio characteristic for a second audio region comprising the second group of the plurality of time frequency bins;

normalize the audio region to generate normalized energy values, wherein normalizing the audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the group of the plurality of time-frequency bins based on the determined audio characteristic associated with the audio region;

normalize the second audio region to generate second normalized energy values, wherein normalizing the second audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the second group of the plurality of time-frequency bins based on the determined second audio characteristic associated with the second audio region;

select one of the normalized energy values, wherein selecting one of the normalized energy values comprises:

determining a category of the audio signal;

weighing each of the time-frequency bins of the group of the plurality of time-frequency bins based on the determined category of the audio signal; and

weighing the selecting of the one of the normalized energy values by the category of the audio signal; and

generate a fingerprint of the audio signal using the selected one of the normalized energy values.

12. The non-transitory computer readable storage medium of claim 11 , wherein the transformation of the audio signal into the frequency domain includes performing a fast Fourier transform of the audio signal.

13. The non-transitory computer readable storage medium of claim 11 , wherein the category of the audio signal includes at least one of music, human speech, sound effects, or advertisement.

14. An apparatus comprising:

at least one memory;

programmable circuitry; and

instructions to cause the programmable circuitry to:

transform an audio signal into a frequency domain, the transformed audio signal including a plurality of time-frequency bins, each of the time-frequency bins corresponding to an intersection of a frequency bin and a time bin and contains a portion of the audio signal;

select a first time-frequency bin;

determine a group of the plurality of time-frequency bins based on the first time-frequency bin and time-frequency bins within a pre-defined distance of the first time-frequency bin;

determine an audio characteristic for an audio region comprising the group of the plurality of time-frequency bins, wherein the determined audio characteristic for the audio region includes at least one of: (i) a mean energy value; (ii) a mode energy value (iii) an average power value; (iv) a mode power value; or (v) a mean amplitude of the group of the plurality of time-frequency bins;

select a second time-frequency bin;

determine a second group of the plurality of time-frequency bins based on the second time-frequency bin and time-frequency bins within a pre-defined distance of the second time-frequency bin, wherein at least a portion of the group of time-frequency bins overlaps at least a portion of the second group of time-frequency bins;

determine a second audio characteristic for a second audio region comprising the second group of the plurality of time frequency bins;

normalize the audio region to generate normalized energy values, wherein normalizing the audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the group of the plurality of time-frequency bins based on the determined audio characteristic associated with the audio region;

normalize the second audio region to generate second normalized energy values, wherein normalizing the second audio region comprises normalizing each portion of the audio signal of each time-frequency bin of the second group of the plurality of time-frequency bins based on the determined second audio characteristic associated with the second audio region;

select one of the normalized energy values, wherein selecting one of the normalized energy values comprises:

determining a category of the audio signal;

weighing each of the time-frequency bins of the group of the plurality of time-frequency bins based on the determined category of the audio signal; and

weighing the selecting of the one of the normalized energy values by the category of the audio signal; and

generate a fingerprint of the audio signal using the selected one of the normalized energy values.

15. The apparatus of claim 14 , wherein the transformation of the audio signal into the frequency domain includes performing a fast Fourier transform of the audio signal.

16. The apparatus of claim 14 , wherein the category of the audio signal includes at least one of music, human speech, sound effects, or advertisement.

17. The apparatus of claim 14 , wherein each time-frequency bin of the plurality of time-frequency bins is a unique combination of (1) a time period of the transformed audio signal and (2) a frequency bin of the transformed audio signal.

Assignments (8)
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
SECURITY INTEREST Recorded May 8, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: ARES CAPITAL CORPORATION
Reel/Frame 063574/0632 →
SECURITY INTEREST Recorded Apr 28, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: CITIBANK, N.A.
Reel/Frame 063561/0381 →
SECURITY AGREEMENT Recorded Jan 31, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: BANK OF AMERICA, N.A.
Reel/Frame 063560/0547 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 4, 2020
From: COOVER, ROBERT; RAFII, ZAFAR
To: GRACENOTE, INC.
Reel/Frame 051713/0782 →
Priority Claims (1)
FR 1858041 · Sep 7, 2018 · national
Continuity (1)
Related Publication 20200082835A1 · Mar 12, 2020
References Cited (47)
US 9202472B1 · Sharifi · 2015 [cited by examiner]
US 9299364B1 · Pereira · 2016 [cited by examiner]
US 9390719B1 · Roblek · 2016 [cited by examiner]
US 20030086341A1 · Wells et al. · 2003 [cited by applicant]
US 20060020958A1 · Allamanche et al. · 2006 [cited by applicant]
US 20080215651A1 · Sawada et al. · 2008 [cited by applicant]
US 20090052784A1 · Covell et al. · 2009 [cited by applicant]
US 20110035035A1 · Khan et al. · 2011 [cited by applicant]
US 20110064244A1 · Eisenberg et al. · 2011 [cited by applicant]
US 20110261257A1 · Terry · 2011 [cited by examiner]
US 20120103166A1 · Shibuya et al. · 2012 [cited by applicant]
US 20130279704A1 · Crockett · 2013 [cited by applicant]
US 20140114456A1 · Stavropoulos et al. · 2014 [cited by applicant]
US 20140180674A1 · Neuhauser et al. · 2014 [cited by applicant]
US 20140310006A1 · Anguera Miro · 2014 [cited by examiner]
US 20160148620A1 · Bilobrov · 2016 [cited by applicant]
US 20160247512A1 · Duong et al. · 2016 [cited by applicant]
US 20170365276A1 · Greene · 2017 [cited by applicant]
US 20190130032A1 · Donier et al. · 2019 [cited by applicant]
US 20190139557A1 · Hodgson et al. · 2019 [cited by applicant]
US 20210157838A1 · Berrian et al. · 2021 [cited by applicant]
JP 2006505821 · 2006 [cited by applicant]
JP 2016518663 · 2016 [cited by applicant]
KR 1020130055115A · 2013 [cited by applicant]
KR 20170027648 · 2017 [cited by applicant]
KR 20180135464 · 2018 [cited by applicant]
Institut National De La Propriete Industrielle, “Written Communcation,” mailed Jan. 21, 2019 in connection with French Patent Application No. FR1858041 (2 pages). [cited by applicant]
Institut National De La Propriete Industrielle, “Preliminary Search Report,” mailed Jul. 26, 2019 in connection with French Patent Application No. FR1858041 (9 pages). [cited by applicant]
Wooram Son et al., “Sub-fingerprint Masking for a Robust Audio Fingerprinting System in a Real-noise Environment for Portable Consumer Devices,” 2010 Digest of Technical Papers/International Conference on Consumer Elect… [cited by applicant]
International Searching Authority, “International Search Report,” mailed Dec. 23, 2019 in connection with International Patent Application No. PCT/US2019/049953, 3 pages. [cited by applicant]
International Searching Authority, “Written Opinion,” mailed Dec. 23, 2019 in connection with International Patent Application No. PCT/US2019/049953, 5 pages. [cited by applicant]
International Bureau, “International Preliminary Report on Patentability,” mailed Mar. 9, 2021 in connection with International Patent Application No. PCT/US2019/049953, 7 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Rules 161(2) and 162 EPC,” issued in connection with European Patent Application No. 19857365.1, dated Apr. 14, 2021, 3 pages. [cited by applicant]
Institut National De La Propriete Industrielle, “Notice of Intention to Grant,” issued Feb. 11, 2021 in connection with French patent application No. 18 58041, 2 pages. (English summary included). [cited by applicant]
Canadian Intellectual Property Office, “Search Report”, issued in connection with Canadian Patent Application No. 3,111,800 on Mar. 7, 2022, 4 pages. [cited by applicant]
Australian Intellectual Property Office, “Examination Report”, issued in connection with Australian Patent Application No. 2019335404 on Jan. 27, 2022, 3 pages. [cited by applicant]
International Searching Authority, “Written Opinion of the International Searching Authority”, issued in connection with International Patent Application No. PCT/US2020/061077 on Mar. 5, 2021, 3 pages. [cited by applicant]
International Searching Authority, “International Search Report” issued in connection with International Patent Application No. PCT/US2020/061077 on Mar. 5, 2021, 3 pages. [cited by applicant]
International Searching Authority, “Written Opinion of the International Searching Authority”, issued in connection with International Patent Application No. PCT/US2022/015442 on May 18, 2022, 3 pages. [cited by applicant]
International Searching Authority, “International Search Report”, issued in connection with International Patent Application No. PCT/US2022/015442 on May 18, 2022, 4 pages. [cited by applicant]
European Patent Office, “Communication of of European Publication Number and Information on the Application of Article 67(3) EPC”, issued in connection with European Patent Application No. 20891501.7 on Sep. 7, 2022, 1 … [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability”, issued in connection with International Patent Application No. PCT/US2020/061077 on May 17, 2022, 4 pages. [cited by applicant]
Japanese Patent Office, “Notice for Reasons of Rejection”, issued in connection with Japanese Patent Application No. 21-512712 on Apr. 26, 2022, 6 pages. [cited by applicant]
European Patent Office, “Extended Search Report”, issued in connection with European Patent Application No. 19857365.1 on Jun. 7, 2022, 8 pages. [cited by applicant]
European Patent Office, “Communication Pursuant to Rule 161(2) and 162 EPC”, issued in connection with European Patent Application No. 20891501.7 on Jul. 5, 2022, 3 pages. [cited by applicant]
IP Australia, “Notice of Acceptance for Patent Application”, issued in connection with Australian Patent Application No. 2019335404 on Aug. 10, 2022, 3 pages. [cited by applicant]
European Patent Office, “Communication Pursuant to Rules 70(2) and 70a(2) EPC”, issued in connection with European Patent Application No. 19857365.1 on Jun. 24, 2022, 1 page. [cited by applicant]