IP Library Granted Patent US 10,984,296
Granted Patent B2
US 10,984,296 · App. 16/418,649 · Granted Apr 20, 2021

Methods and apparatus for identifying objects depicted in a video using extracted video frames in combination with a reverse image search engine

Inventor: Jan Besehanic (Tampa, FL)
Assignee: THE NIELSEN COMPANY (US), LLC
G06K9/78G06F11/3476G06F16/40G06F16/50G06F16/58G06F16/583G06F16/78G06F16/7837G06K9/00624G06K9/18G06K9/325G06K2209/25
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,984,296
App. No.
16/418,649
Granted
Apr 20, 2021
Kind
B2
Abstract

Example apparatus disclosed herein are to: capture screenshots of search results pages returned from a reverse image search engine for respective ones of image frames of a first video processed by the reverse image search engine; generate a plurality of base query records corresponding to ones of the screenshots that have respective textual information associated with a base search term, the respective ones of the base query records including at least portions of the respective textual information from the corresponding screenshots; perform a frequency analysis on the textual information included in the base query records to determine an object search term; generate a plurality of object query records corresponding to respective ones of the base query records that have respective textual information associated with the object search term; and identify a first object depicted in the first video based on at least one criteria applied to the object query records.

Claims (37)

1. An apparatus to identify objects depicted in videos, the apparatus comprising:

memory including machine readable instructions; and

processor circuitry to execute the instructions to:

capture screenshots of search results pages returned via a network from a reverse image search engine, respective ones of the search results pages corresponding to respective ones of image frames of a first video processed by the reverse image search engine;

generate a plurality of base query records corresponding to respective ones of the screenshots that have respective textual information matching a base search term, the respective ones of the base query records including the base search term and at least portions of the respective textual information from the corresponding screenshots;

determine an object search term based on a frequency analysis of the textual information included in the base query records, the frequency analysis excluding the base search term from the base query records;

generate a plurality of object query records corresponding to respective ones of the base query records that have respective textual information matching the object search term; and

identify a first object depicted in the first video based on at least one criteria applied to the object query records.

2. The apparatus as defined in claim 1 , wherein the respective ones of the image frames are accessible from an image repository by the reverse image search engine based on resource locators associated with the image frames, the reverse image search engine in communication with the image repository via the network.

3. The apparatus as defined in claim 1 , wherein the frequency analysis includes at least one of a word count analysis, a phrase count analysis, or a string comparison analysis.

4. The apparatus as defined in claim 1 , wherein the base search term is associated with at least one of a word or a phrase included in the respective ones of the search results pages returned by the reverse image search engine.

5. The apparatus as defined in claim 1 , wherein the base search term is associated with at least one of a position or a threshold relative to a configuration of a layout included in the respective ones of the search results pages returned by the reverse image search engine.

6. The apparatus as defined in claim 1 , wherein the respective portions of the respective textual information from the corresponding screenshots at least one of follow the base search term in the corresponding screenshots, are on a same line as the base search term in the corresponding screenshots, or are part of a same string as the base search term in the corresponding screenshots.

7. The apparatus as defined in claim 1 , wherein the at least one criteria includes a threshold based on a first number of the object query records including the object search term.

8. The apparatus as defined in claim 1 , wherein the at least one criteria includes a threshold based on a first number of the object query records including the object search term within a first time period, the first time period based on a sampling rate at which the image frames were obtained from the first video.

9. The apparatus as defined in claim 1 , wherein the at least one criteria includes a threshold based on a first consecutive number of the object query records including the object search term.

10. The apparatus as defined in claim 1 , wherein the criteria includes a threshold based on a first percentage of a number of object query records including the object search term relative to a number of base query records including the base search term.

11. A method to identify objects depicted in videos, the method comprising:

capturing, by executing an instruction with at least one processor, screenshots of search results pages returned via a network from a reverse image search engine, respective ones of the search results pages corresponding to respective ones of image frames of a first video processed by the reverse image search engine;

generating, by executing an instruction with the at least one processor, a plurality of base query records corresponding to respective ones of the screenshots that have respective textual information matching a base search term, the respective ones of the base query records including the base search term and at least portions of the respective textual information from the corresponding screenshots;

determining, by executing an instruction with the at least one processor, an object search term based on a frequency analysis of the textual information included in the base query records, the frequency analysis excluding the base search term from the base query records;

generating, by executing an instruction with the at least one processor, a plurality of object query records corresponding to respective ones of the base query records that have respective textual information matching the object search term; and

identifying, by executing an instruction with the at least one processor, a first object depicted in the first video based on at least one criteria applied to the object query records.

12. The method as defined in claim 11 , wherein the frequency analysis includes at least one of a word count analysis, a phrase count analysis, or a string comparison analysis.

13. The method as defined in claim 11 , wherein the base search term is associated with at least one of a word or a phrase included in the respective ones of the search results pages returned by the reverse image search engine.

14. The method as defined in claim 11 , wherein the base search term is associated with at least one of a position or a threshold relative to a configuration of a layout included in the respective ones of the search results pages returned by the reverse image search engine.

15. The method as defined in claim 11 , wherein the respective portions of the respective textual information from the corresponding screenshots at least one of follow the base search term in the corresponding screenshots, are on a same line as the base search term in the corresponding screenshots, or are part of a same string as the base search term in the corresponding screenshots.

16. A non-transitory computer readable medium comprising computer readable instructions that, when executed, cause a processor to at least:

capture screenshots of search results pages returned via a network from a reverse image search engine, respective ones of the search results pages corresponding to respective ones of image frames of a first video processed by the reverse image search engine;

generate a plurality of base query records corresponding to respective ones of the screenshots that have respective textual information matching a base search term, the respective ones of the base query records including the base search term and at least portions of the respective textual information from the corresponding screenshots;

determine an object search term based on a frequency analysis of the textual information included in the base query records, the frequency analysis excluding the base search term from the base query records;

generate a plurality of object query records corresponding to respective ones of the base query records that have respective textual information matching the object search term; and

identify a first object depicted in the first video based on at least one criteria applied to the object query records.

17. The non-transitory computer readable medium as defined in claim 16 , wherein the frequency analysis includes at least one of a word count analysis, a phrase count analysis, or a string comparison analysis.

18. The non-transitory computer readable medium as defined in claim 16 , wherein the base search term is associated with at least one of a word or a phrase included in the respective ones of the search results pages returned by the reverse image search engine.

19. The non-transitory computer readable medium as defined in claim 16 , wherein the base search term is associated with at least one of a position or a threshold relative to a configuration of a layout included in the respective ones of the search results pages returned by the reverse image search engine.

20. The non-transitory computer readable medium as defined in claim 16 , wherein the respective portions of the respective textual information from the corresponding screenshots at least one of follow the base search term in the corresponding screenshots, are on a same line as the base search term in the corresponding screenshots, or are part of a same string as the base search term in the corresponding screenshots.

Assignments (8)
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
SECURITY INTEREST Recorded May 8, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: ARES CAPITAL CORPORATION
Reel/Frame 063574/0632 →
SECURITY INTEREST Recorded Apr 28, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: CITIBANK, N.A.
Reel/Frame 063561/0381 →
SECURITY AGREEMENT Recorded Jan 31, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: BANK OF AMERICA, N.A.
Reel/Frame 063560/0547 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 22, 2019
From: BESEHANIC, JAN
To: THE NIELSEN COMPANY (US), LLC
Reel/Frame 049255/0402 →
Cited By (1)
US 12,705,278