IP Library › Granted Patent US 12,217,527
Granted Patent B2
US 12,217,527 · App. 18/363,378 · Granted Feb 4, 2025

Methods and apparatus to determine an audience composition based on voice recognition, thermal imaging, and facial recognition

Inventors: John T. LiVoti (Clearwater, FL); Stanley Wellington Woodruff (Palm Harbor, FL)
Assignee: The Nielsen Company (US), LLC
G06V40/10G06V10/751G10L25/51G10L25/78H04N21/44218
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,217,527
App. No.
18/363,378
Granted
Feb 4, 2025
Kind
B2
Abstract

Methods, apparatus, systems and articles of manufacture are disclosed. An example apparatus includes an audio detector to determine a first audience count based on signatures of audio data captured in the media environment, a thermal image detector to determine a heat blob count based on a frame of thermal image data captured in the media environment, and an audience image detector to identify at least one audience member based on a comparison of a frame of audience image data with a library of reference audience images, the audience image detector to perform the comparison in response to the first audience count not matching the heat blob count.

Claims (59)

1. An apparatus comprising:

an audio sensor to generate audio monitoring data during presentation of media presented by a media presentation device in a media environment;

a thermal image sensor to generate thermal image data captured in the media environment;

a light image sensor to generate audience image data of a selected field of view of the media environment; and

at least one processor to execute computer readable instructions to perform a set of operations including:

based on the generated thermal image data, determining a thermal-indicated audience count that corresponds to a number of heat blobs within the generated thermal image data;

comparing the thermal-indicated audience count with a previously obtained audience count for the media environment;

determining, based on comparing the thermal-indicated audience count with the previously obtained audience count, that the thermal-indicated audience count differs from the previously obtained audience count;

based on the generated thermal image data, determining a position, within the generated thermal image data, that corresponds to an audience member;

defining the selected field of view of the media environment based on the determined position;

responsive to determining that the thermal-indicated audience count differs from the previously obtained audience count, controlling the light image sensor to generate the audience image data of the selected field of view; and

identifying an audience member based on a comparison of a frame of the audience image data with a library of reference audience images.

2. The apparatus of claim 1 , further including a user input receiver to obtain an input from one or more audience members to indicate the previously obtained audience count.

3. The apparatus of claim 2 , further including a user interface to prompt the one or more audience members to indicate the previously obtained audience count, and wherein the set of operations further include prompting the one or more audience members to indicate a current audience count following a threshold time period after identifying the audience member.

4. The apparatus of claim 3 , wherein the threshold time period is reset upon at least one of a subsequent identification of another audience member, or a subsequent re-identification of the audience member.

5. The apparatus of claim 1 , further including a network communication interface, and wherein the operations further include transmitting, via the network communication interface, audience meter data including data indicative of at least the identified audience member.

6. The apparatus of claim 5 , wherein the audio monitoring data is indicative of a media identifier corresponding to the media presented by the media presentation device, and wherein the transmitted audience meter data further includes the media identifier.

7. The apparatus of claim 1 , wherein the operations further includes:

determining a number of distinct voices represented within the audio monitoring data; and

determining an audio-indicated audience count based on the determined number of distinct voices.

8. A method performed by computing system including at least one processor, the method comprising:

obtaining thermal image data of a media environment during presentation of media presented by a media presentation device in the media environment;

determining, based on the thermal image data, a thermal-indicated audience count that corresponds to a number of heat blobs within the generated thermal image data;

comparing the thermal-indicated audience count with a previously obtained audience count for the media environment;

determining, based on comparing the thermal-indicated audience count with the previously obtained audience count, that the thermal-indicated audience count differs from the previously obtained audience count;

determining, based on the thermal image data, a position, within the generated thermal image data, that corresponds to an audience member;

defining a field of view of the media environment based on the determined position;

responsive to determining that the thermal-indicated audience count differs from the previously obtained audience count, controlling a light image sensor to generate audience image data of the defined field of view; and

identifying an audience member based on a comparison of a frame of the audience image data with a library of reference audience images.

9. The method of claim 8 , further including obtaining an input, via a user interface, indicating the previously obtained audience count.

10. The method of claim 9 , further including:

prompting, via the user interface, one or more audience members to indicate the previously obtained audience count, wherein obtaining the input indicating the previously obtained audience count is responsive to prompting the one or more audience members; and

subsequently prompting the one or more audience members to indicate a current audience count following a threshold time period after identifying the audience member.

11. The method of claim 10 , further including resetting the threshold time period upon at least one of a subsequent identification of another audience member, or a subsequent re-identification of the audience member.

12. The method of claim 8 , further including transmitting, via a network communication interface, audience meter data including data indicative of at least the identified audience member.

13. The method of claim 8 , further including: obtaining audio monitoring data, via an audio sensor, during presentation of the media presented by the media presentation device, wherein the audio monitoring data is indicative of a media identifier corresponding to the media presented by the media presentation device.

14. The method of claim 13 , further including:

determining a number of distinct voices represented within the audio monitoring data; and

determining an audio-indicated audience count based on the determined number of distinct voices.

15. A non-transitory computer readable storage medium having stored therein instructions that, upon execution by at least one processor, cause performance of:

obtaining thermal image data of a media environment during presentation of media presented by a media presentation device in the media environment;

determining, based on the thermal image data, a thermal-indicated audience count that corresponds to a number of heat blobs within the generated thermal image data;

comparing the thermal-indicated audience count with a previously obtained audience count for the media environment;

determining, based on comparing the thermal-indicated audience count with the previously obtained audience count, that the thermal-indicated audience count differs from the previously obtained audience count;

determining, based on the thermal image data, a position, within the generated thermal image data, that corresponds to an audience member;

defining a field of view of the media environment based on the determined position;

responsive to determining that the thermal-indicated audience count differs from the previously obtained audience count, controlling a light image sensor to generate audience image data of the defined field of view; and

identifying an audience member based on a comparison of a frame of the audience image data with a library of reference audience images.

16. The non-transitory computer readable storage medium of claim 15 , wherein the instructions further cause, upon execution by the at least one processor, performance of:

obtaining an input, via a user interface, indicating the previously obtained audience count.

17. The non-transitory computer readable storage medium of claim 16 , wherein the instructions further cause, upon execution by the at least one processor, performance of:

prompting, via the user interface, one or more audience members to indicate the previously obtained audience count, wherein obtaining the input indicating the previously obtained audience count is responsive to prompting the one or more audience members; and

subsequently prompting the one or more audience members to indicate a current audience count following a threshold time period after identifying the audience member.

18. The non-transitory computer readable storage medium of claim 17 , wherein the instructions further cause, upon execution by the at least one processor, performance of resetting the threshold time period upon at least one of a subsequent identification of another audience member, or a subsequent re-identification of the audience member.

19. The non-transitory computer readable storage medium of claim 15 , wherein the instructions further cause, upon execution by the at least one processor, performance of transmitting, via a network communication interface, audience meter data including data indicative of at least the identified audience member.

20. The non-transitory computer readable storage medium of claim 15 , wherein the instructions further cause, upon execution by the at least one processor, performance of:

obtaining audio monitoring data, via an audio sensor, during presentation of the media presented by the media presentation device, wherein the audio monitoring data is indicative of a media identifier corresponding to the media presented by the media presentation device;

determining a number of distinct voices represented within the audio monitoring data; and

determining an audio-indicated audience count based on the determined number of distinct voices.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 1, 2023
From: LIVOTI, JOHN T.; WOODRUFF, STANLEY WELLINGTON
To: THE NIELSEN COMPANY (US), LLC
Reel/Frame 064454/0406 →
Continuity (2)
Continuation 16998814 · Aug 20, 2020
Related Publication 20230410547A1 · Dec 21, 2023
References Cited (32)
US 5481294A · Thomas et al. · 1996 [cited by applicant]
US 6571865B1 · Shi et al. · 2003 [cited by applicant]
US 6849199B2 · Vickers, Jr. et al. · 2005 [cited by applicant]
US 7839291B1 · Richards · 2010 [cited by examiner]
US 9489646B2 · Puttabasappa et al. · 2016 [cited by applicant]
US 11553247B2 · Woodruff et al. · 2023 [cited by applicant]
US 11595723B2 · LiVoti et al. · 2023 [cited by applicant]
US 20030012425A1 · Suzuki et al. · 2003 [cited by applicant]
US 20040059438A1 · Sherlock · 2004 [cited by applicant]
US 20080159591A1 · Ruedin · 2008 [cited by applicant]
US 20110215932A1 · Daniel · 2011 [cited by examiner]
US 20120023976A1 · Kim et al. · 2012 [cited by applicant]
US 20130083965A1 · Joung · 2013 [cited by examiner]
US 20130132271A1 · Daniel · 2013 [cited by applicant]
US 20150092019A1 · Asano et al. · 2015 [cited by applicant]
US 20150199008A1 · Kim et al. · 2015 [cited by applicant]
US 20160066034A1 · Hicks · 2016 [cited by examiner]
US 20170147885A1 · Aggarwal et al. · 2017 [cited by applicant]
US 20180240251A1 · Kitagawa · 2018 [cited by applicant]
US 20180253221A1 · Cho et al. · 2018 [cited by applicant]
US 20180295420A1 · Rumreich et al. · 2018 [cited by applicant]
US 20180356968A1 · Chiu · 2018 [cited by applicant]
US 20190199610A1 · Kuenzi · 2019 [cited by examiner]
US 20190330214A1 · Miyazaki et al. · 2019 [cited by applicant]
US 20210195045A1 · Kim et al. · 2021 [cited by applicant]
US 20210374394A1 · Livoti · 2021 [cited by examiner]
US 20210397852A1 · Tabor · 2021 [cited by applicant]
US 20220058382A1 · LiVoti et al. · 2022 [cited by applicant]
US 20220103022A1 · Bell et al. · 2022 [cited by applicant]
WO 2021247706A1 · 2021 [cited by applicant]
United States Patent and Trademark Office, “Non-Final Rejection,” in connection with U.S. Appl. No. 17/850,726, issued Apr. 6, 2023, 8 pages. [cited by applicant]
USPTO, Non-Final Office action dated Sep. 26, 2024 for U.S. Appl. No. 18/610,779, 8 pages. [cited by applicant]