IP Library Granted Patent US 10,997,424
Granted Patent B2
US 10,997,424 · App. 16/257,339 · Granted May 4, 2021

Methods and systems for sport data extraction

Inventors: Jeffrey Scott (Oakland, CA); Markus Kurt Peter Cremer (Orinda, CA); Nishit Umesh Parekh (Woodland Hills, CA); Dewey Ho Lee (Berkeley, CA)
Assignee: Gracenote, inc.
G06K9/00724G06K9/325G06T7/13H04N21/2187
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,997,424
App. No.
16/257,339
Granted
May 4, 2021
Kind
B2
Abstract

A computing system engages in digital image processing of received video frames to generate sport data that indicates a score and/or a time associated with a sport event. The digital image processing includes: (i) identifying a first frame region of the video frames based on the first frame region depicting a scoreboard; (ii) executing a first procedure that analyzes the identified first frame region to detect, within the identified first frame region, second frame region(s) based on the second frame region(s) depicting text of the scoreboard; (iii) in response to detecting the second frame region(s), executing a second procedure to recognize the text in at least one of the second frame region(s); and (iv) based at least on the recognizing of the text, generating the sport data. In response to completing the digital image processing, the computing system then carries out an action based on the generated sport data.

Claims (74)

1. A method comprising:

receiving, by a computing system, video frames that depict a sport event, wherein the depicted sport event includes a depiction of a sport score associated with the sport event, and wherein the computing system comprises a processing unit;

engaging, by the computing system, in digital image processing of the received video frames to automatically generate sport data that indicates the sport score, wherein the digital image processing comprises:

(i) identifying a first frame region of the video frames based on the first frame region depicting a scoreboard,

(ii) executing a first machine-driven pattern recognition procedure that includes analyzing the identified first frame region to detect, within the identified first frame region, one or more second frame regions based on the second frame regions depicting text of the scoreboard,

(iii) in response to detecting the second frame regions, executing a second machine-driven pattern recognition procedure to recognize the text in at least one of the one or more second frame regions, and

(iv) based at least on the recognizing of the text, generating the sport data that indicates the sport score; and

in response to completing the digital image processing, carrying out, by the computing system, an action based on the generated sport data.

2. The method of claim 1 , wherein receiving the video frames comprises receiving a real-time broadcast feed that includes the video frames.

3. The method of claim 2 , wherein engaging in the digital image processing comprises engaging in the digital image processing while receiving the real-time broadcast feed.

4. The method of claim 1 , further comprising:

receiving, by the computing system, electronic program guide (EPG) data that indicates event types respectively associated with a plurality of broadcast feeds, wherein a particular broadcast feed of the plurality includes the video frames; and

based on the received EPG data, making a determination, by the computing system, that the particular broadcast feed is associated with the sport event,

wherein engaging in the digital image processing comprises triggering the digital image processing in response to making the determination.

5. The method of claim 4 , wherein the EPG data also indicates times respectively associated with scheduling of the plurality of broadcast feeds, the method further comprising:

based on the received EPG data, determining, by the computing system, a particular time associated with scheduling of the particular broadcast feed that includes the video frames, and

wherein engaging in the digital image processing comprises, based on the determining of the particular time, triggering the digital image processing at the determined particular time.

6. The method of claim 1 , further comprising:

receiving, by the computing system, a request to trigger the digital image processing,

wherein engaging in the digital image processing comprises triggering the digital image processing in response the received request.

7. The method of claim 1 , wherein generating sport data that indicates the sport score comprises generating sport data that indicates one or more of the following: (i) a first team name, (ii) a second team name, (iii) a first score associated with the first team name, and (iv) a second score associated with the second team name.

8. The method of claim 1 , further comprising:

receiving, by the computing system, information indicating the first frame region that depicts the scoreboard,

wherein identifying the first frame region is based on the received information indicating the first frame region.

9. The method of claim 1 , wherein identifying the first frame region comprises executing a third machine-driven pattern recognition procedure that analyzes one or more of the video frames to identify the first frame region depicting the scoreboard.

10. The method of claim 9 ,

wherein the third machine-driven pattern recognition procedure analyzes at least a particular one of the video frames to identify the first frame region depicting the scoreboard, and

wherein the first machine-driven pattern recognition procedure analyzes the first frame region within the particular video frame to detect the one or more second frame regions that depict text of the scoreboard.

11. The method of claim 9 ,

wherein the first frame region appears in at least a first and a second of the video frames,

wherein the third machine-driven pattern recognition procedure analyzes at least the first video frame to identify the first frame region depicting the scoreboard, and

wherein the first machine-driven pattern recognition procedure analyzes the first frame region within the second video frame to detect the one or more second frame regions that depict text of the scoreboard.

12. The method of claim 1 , wherein the digital image processing further comprises recording one or more of (i) a location of the identified first frame region within the video frames, (ii) an area within the video frames that the identified first frame region encompasses, (iii) a location of the detected one or more second frame regions within the video frames, and (iv) an area within the video frames that the detected one or more second frame regions encompass.

13. The method of claim 1 ,

wherein the video frames also depict a game time associated with the sport event,

wherein engaging in the digital image processing is also to automatically generate sport data that indicates the game time,

wherein the digital image processing further comprises, based at least on the recognizing of the text, generating the sport data that indicates the game time, and

wherein the action is also based on the generated sport data that indicates the game time.

14. The method of claim 1 , wherein carrying out the action comprises one or more of the following:

(i) storing the generated sport data in a data storage device,

(ii) causing presentation of the sport score indicated by the generated sport data,

(iii) outputting a report related to the generated sport data,

(iv) updating information in a user-account based on the generated sport data, and

(v) causing presentation of supplemental content based on the generated sport data.

15. A method comprising:

receiving, by a computing system, video frames that depict a sport event, wherein the depicted sport event includes a depiction of a game time associated with the sport event, and wherein the computing system comprises a processing unit;

engaging, by the computing system, in digital image processing of the received video frames to automatically generate sport data that indicates the game time, wherein the digital image processing comprises:

(i) identifying a first frame region of the video frames based on the first frame region depicting a scoreboard,

(ii) executing a first machine-driven pattern recognition procedure that includes analyzing the identified first frame region to detect, within the identified first frame region, one or more second frame regions based on the second frame regions depicting text of the scoreboard,

(iii) in response to detecting the second frame regions, executing a second machine-driven pattern recognition procedure to recognize the text in at least one of the one or more second frame regions, and

(iv) based at least on the recognizing of the text, generating the sport data that indicates the game time; and

in response to completing the digital image processing, carrying out, by the computing system, an action based on the generated sport data.

16. The method of claim 15 , wherein receiving the video frames comprises receiving a real-time broadcast feed that includes the video frames, and wherein engaging in the digital image processing comprises engaging in the digital image processing while receiving the real-time broadcast feed.

17. The method of claim 15 , further comprising:

receiving, by the computing system, information indicating the first frame region that depicts the scoreboard,

wherein identifying the first frame region is based on the received information indicating the first frame region.

18. The method of claim 15 , wherein identifying the first frame region comprises executing a third machine-driven pattern recognition procedure that analyzes one or more of the video frames to identify the first frame region depicting the scoreboard.

19. The method of claim 15 , wherein carrying out the action comprises one or more of the following:

(i) storing the generated sport data in a data storage device,

(ii) causing presentation of the game time indicated by the generated sport data,

(iii) outputting a report related to the generated sport data,

(iv) updating information in a user-account based on the generated sport data, and

(v) causing presentation of supplemental content based on the generated sport data.

20. A computing system comprising:

a processing unit;

non-transitory data storage; and

program instructions stored in the non-transitory data storage and executable by the processing unit to:

receive video frames that depict a sport event, wherein the depicted sport event includes respective depictions of one or more of a sport score and a game time associated with the sport event,

engage in digital image processing of the received video frames to automatically generate sport data that indicates one or more of the sport score and the game time, wherein the digital image processing comprises:

(i) identifying a first frame region of the video frames based on the first frame region depicting a scoreboard,

(ii) executing a first machine-driven pattern recognition procedure that includes analyzing the identified first frame region to detect, within the identified first frame region, one or more second frame regions based on the second frame regions depicting text of the scoreboard,

(iii) in response to detecting the second frame regions, executing a second machine-driven pattern recognition procedure to recognize the text in at least one of the one or more second frame regions, and

(iv) based at least on the recognizing of the text, generating the sport data that indicates one or more of the sport score and the game time, and

in response to completing the digital image processing, carry out an action based on the generated sport data.

Assignments (8)
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
SECURITY INTEREST Recorded May 8, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: ARES CAPITAL CORPORATION
Reel/Frame 063574/0632 →
SECURITY INTEREST Recorded Apr 28, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: CITIBANK, N.A.
Reel/Frame 063561/0381 →
SECURITY AGREEMENT Recorded Jan 31, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: BANK OF AMERICA, N.A.
Reel/Frame 063560/0547 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 25, 2019
From: SCOTT, JEFFREY; CREMER, MARKUS KURT PETER; PAREKH, NISHIT UMESH; LEE, DEWEY HO
To: GRACENOTE, INC.
Reel/Frame 048134/0343 →
Cited By (1)
US 12,300,005