IP Library Granted Patent US 10,331,984
Granted Patent B2
US 10,331,984 · App. 16/051,224 · Granted Jun 25, 2019

Methods and apparatus for identifying objects depicted in a video using extracted video frames in combination with a reverse image search engine

Inventor: Jan Besehanic (Tampa, FL)
Assignee: THE NIELSEN COMPANY (US), LLC
G06K9/78G06F11/3476G06F16/50G06F16/58G06F16/583G06F16/78G06F16/7837G06K9/00624G06K9/18G06K9/325G06K2209/25
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,331,984
App. No.
16/051,224
Granted
Jun 25, 2019
Kind
B2
Abstract

Methods and apparatus are disclosed for identifying objects depicted in videos. An image repository stores image frames of a video. Respective ones of the image frames are accessed from the image repository by a reverse image search engine in communication with the image repository via a network. A screen capturer captures screenshots of search results pages returned via the network from the reverse image search engine, thereby converting dynamic data included in respective ones of the search results pages into corresponding static data included in respective ones of the screenshots. The search results pages correspond to the respective ones of the image frames. A text identifier identifies textual information presented in the respective ones of the screenshots. An object identifier identifies an object depicted in the video based on at least one criteria applied to records generated from portions of the textual information determined to be associated with a search term.

Claims (46)

1. An apparatus to identify objects depicted in videos, the apparatus comprising:

an image repository structured to store image frames of a first video, respective ones of the image frames to be accessible from the image repository by a reverse image search engine in communication with the image repository via a network;

a screen capturer structured to capture screenshots of search results pages returned via the network from the reverse image search engine, the screen capturer to convert dynamic data included in respective ones of the search results pages into corresponding static data included in respective ones of the screenshots, the search results pages corresponding to the respective ones of the image frames;

a text identifier structured to identify textual information presented in the respective ones of the screenshots;

a base query record generator structured to generate a plurality of base query records corresponding to respective ones of the screenshots that have respective textual information satisfying a base search term, the respective ones of the base query records including the base search term and portions of the textual information that are associated with the base search term in the corresponding screenshots;

an object query record generator structured to:

perform a frequency analysis on the respective ones of the base query records, the frequency analysis including at least one of a word count analysis, a phrase count analysis, or a string comparison analysis, the frequency analysis to filter out the base search term from the respective ones of the base query records;

determine an object search term based on a result of the frequency analysis; and

generate a plurality of object query records corresponding to respective ones of the base query records that satisfy the object search term; and

an object identifier structured to identify a first object depicted in the first video based on at least one criteria applied to the object query records, wherein at least one of the screen capturer, the text identifier, the base query record generator, the object query record generator, or the object identifier is implemented via a logic circuit.

2. The apparatus as defined in claim 1 , wherein the respective ones of the image frames are to be accessible from the image repository by the reverse image search engine based on resource locators associated with the image frames.

3. The apparatus as defined in claim 1 , wherein the base search term is associated with at least one of a word or a phrase included in the respective ones of the search results pages returned by the reverse image search engine.

4. The apparatus as defined in claim 1 , wherein the base search term is associated with at least one of a position or a threshold relative to a configuration of a layout included in the respective ones of the search results pages returned by the reverse image search engine.

5. The apparatus as defined in claim 1 , wherein the respective portions at least one of follow the base search term in the corresponding screenshots, are on a same line as the base search term in the corresponding screenshots, or are part of a same string as the base search term in the corresponding screenshots.

6. The apparatus as defined in claim 1 , wherein the at least one criteria includes a threshold based on a first number of the object query records including the object search term.

7. The apparatus as defined in claim 1 , wherein the at least one criteria includes a threshold based on a first number of the object query records including the object search term within a first time period, the first time period based on a sampling rate at which the image frames were obtained from the first video.

8. The apparatus as defined in claim 1 , wherein the at least one criteria includes a threshold based on a first consecutive number of the object query records including the object search term.

9. The apparatus as defined in claim 1 , wherein the criteria includes a threshold based on a first percentage of a number of object query records including the object search term relative to a number of base query records including the base search term.

10. A method to identify objects depicted in videos, the method comprising:

storing, by executing an instruction with at least one processor, image frames of a first video in an image repository, respective ones of the image frames being accessible from the image repository by a reverse image search engine in communication with the image repository via a network;

capturing, by executing an instruction with the at least one processor, screenshots of search results pages returned via the network from the reverse image search engine, the capturing of the screenshots including converting dynamic data included in respective ones of the search results pages into corresponding static data included in respective ones of the screenshots, the search results pages corresponding to the respective ones of the image frames;

identifying, by executing an instruction with the at least one processor, textual information presented in the respective ones of the screenshots;

generating, by executing an instruction with the at least one processor, a plurality of base query records corresponding to respective ones of the screenshots that have respective textual information satisfying a base search term, the respective ones of the base query records including the base search term and respective portions of the textual information that are associated with the base search term in the corresponding screenshots;

performing, by executing an instruction with the at least one processor, a frequency analysis on the respective ones of the base query records, the frequency analysis including at least one of a word count analysis, a phrase count analysis, or a string comparison analysis, the frequency analysis filtering out the base search term from the respective ones of the base query records;

determining, by executing an instruction with the at least one processor, an object search term based on a result of the frequency analysis;

generating, by executing an instruction with the at least one processor, a plurality of object query records corresponding to respective ones of the base query records that satisfy the object search term; and

identifying, by executing an instruction with the at least one processor, a first object depicted in the first video based on at least one criteria applied to the object query records.

11. The method as defined in claim 10 , wherein the respective ones of the image frames are accessible from the image repository by the reverse image search engine based on resource locators associated with the image frames.

12. The method as defined in claim 10 , wherein the base search term is associated with at least one of a word or a phrase included in the respective ones of the search results pages returned by the reverse image search engine.

13. The method as defined in claim 10 , wherein the respective portions at least one of follow the base search term in the corresponding screenshots, are on a same line as the base search term in the corresponding screenshots, or are part of a same string as the base search term in the corresponding screenshots.

14. The method as defined in claim 10 , wherein the at least one criteria includes a threshold based on a first number of the object query records including the object search term.

15. The method as defined in claim 10 , wherein the at least one criteria includes a threshold based on a first number of the object query records including the object search term within a first time period, the first time period based on a sampling rate at which the image frames were obtained from the first video.

16. The method as defined in claim 10 , wherein the at least one criteria includes a threshold based on a first consecutive number of the object query records including the object search term.

17. A non-transitory computer readable medium comprising computer readable instructions that, when executed, cause a processor to at least:

store image frames of a first video in an image repository, respective ones of the image frames to be accessible from the image repository by a reverse image search engine in communication with the image repository via a network;

capture screenshots of search results pages returned via the network from the reverse image search engine;

convert dynamic data included in respective ones of the search results pages into corresponding static data included in respective ones of the screenshots, the search results pages corresponding to the respective ones of the image frames;

identify textual information presented in the respective ones of the screenshots;

generate a plurality of base query records corresponding to respective ones of the screenshots that have respective textual information satisfying a base search term, the respective ones of the base query records including the base search term and respective portions of the textual information that are associated with the base search term in the corresponding screenshots;

perform a frequency analysis on the respective ones of the base query records, the frequency analysis including at least one of a word count analysis, a phrase count analysis, or a string comparison analysis, the frequency analysis to filter out the base search term from the respective ones of the base query records;

determine an object search term based on a result of the frequency analysis;

generate a plurality of object query records corresponding to respective ones of the base query records that satisfy the object search term; and

identify a first object depicted in the first video based on at least one criteria applied to the object query records.

18. The non-transitory computer readable medium as defined in claim 17 , wherein the respective ones of the image frames are to be accessible from the image repository by the reverse image search engine based on resource locators associated with the image frames.

19. The non-transitory computer readable medium as defined in claim 17 , wherein the base search term is associated with at least one of a word or a phrase included in the respective ones of the search results pages returned by the reverse image search engine.

20. The non-transitory computer readable medium as defined in claim 17 , wherein the respective portions at least one of follow the base search term in the corresponding screenshots, are on a same line as the base search term in the corresponding screenshots, or are part of a same string as the base search term in the corresponding screenshots.

Assignments (8)
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
SECURITY INTEREST Recorded May 8, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: ARES CAPITAL CORPORATION
Reel/Frame 063574/0632 →
SECURITY INTEREST Recorded Apr 28, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: CITIBANK, N.A.
Reel/Frame 063561/0381 →
SECURITY AGREEMENT Recorded Jan 31, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: BANK OF AMERICA, N.A.
Reel/Frame 063560/0547 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 31, 2018
From: BESEHANIC, JAN
To: THE NIELSEN COMPANY (US), LLC
Reel/Frame 046517/0832 →
Continuity (2)
Continuation 14750552 · Jun 25, 2015
Related Publication 20180336442A1 · Nov 22, 2018