Methods and systems for scoreboard region detection
A computing system automatically detects, in a sequence of video frames, a video frame region that depicts a scoreboard. The video frames of the sequence depict image elements including (i) scoreboard image elements that are unchanging across the video frames of the sequence and (ii) other image elements that change across the video frames of the sequence. Given this, the computing system (a) receives the sequence, (b) engages in an edge-detection process to detect, in the video frames of the sequence, a set of edges of the depicted image elements, (c) identifies a subset of the detected set of edges based on each edge of the subset being unchanging across the video frames of the sequence, and (d) detects, based on the edges of the identified subset, the video frame region that depicts the scoreboard.
1. A method for automatically detecting, in a sequence of video frames, a video frame region that depicts a scoreboard, the method comprising:
receiving, by a computing system, the sequence of video frames, wherein the video frames of the sequence depict image elements including (i) scoreboard image elements that are unchanging across the video frames of the sequence and (ii) other image elements that change across the video frames of the sequence, wherein the computing system comprises a processing unit;
engaging, by the computing system, in an edge-detection process to detect, in the video frames of the sequence, a set of edges of the depicted image elements;
identifying, by the computing system, a subset of the detected set of edges based on each edge of the subset being unchanging across the video frames of the sequence, wherein the edges of the identified subset are candidate edges of the scoreboard image elements;
detecting by the computing system, based on the edges of the identified subset, the video frame region that depicts the scoreboard;
executing, by the computing system, a machine-driven pattern recognition procedure to extract a sport score from the detected video frame region in a given video frame of the sequence;
making a determination, by the computing system, of whether the machine-driven pattern recognition procedure successfully extracted the sport score from the detected video frame region in the given video frame;
if the determination is that the machine-driven pattern recognition procedure successfully extracted the sport score from the detected video frame region in the given video frame, then, responsive to making the determination, the computing system taking a first action that corresponds to the detected video frame region being accurate; and
if the determination is that the machine-driven pattern recognition procedure did not successfully extract the sport score from the detected video frame region in the given video frame, then, responsive to making the determination, the computing system taking a second action that corresponds to the detected video frame region being inaccurate.
2. The method of claim 1 , wherein receiving the sequence comprises receiving a real-time broadcast feed including the sequence, and wherein automatically detecting the video frame region comprises automatically detecting the video frame region while receiving the real-time broadcast feed.
3. The method of claim 1 , wherein receiving the sequence comprises receiving a plurality of video frames that include the sequence, and wherein the plurality of video frames depict a sport-related event, the method further comprising:
determining, by the computing system, the sport-related event depicted by the plurality of video frames;
based on the determined sport-related event, determining, by the computing system, a quantity of video frames to use as basis for automatically detecting the video frame region; and
selecting by the computing system, from among the plurality of video frames, the sequence based on the selected sequence including the determined quantity of video frames.
4. The method of claim 3 , further comprising:
receiving, by the computing system, electronic program guide (EPG) data that indicates event types respectively associated with a plurality of broadcast feeds, wherein a particular broadcast feed, from among the plurality of broadcast feeds, includes the plurality of video frames,
wherein determining the sport-related event depicted by the plurality of video frames comprises, based on the received EPG data, determining the sport-related event associated with the particular broadcast feed.
5. The method of claim 1 , wherein engaging in the edge-detection process to detect the set of edges comprises engaging in canny edge detection to detect the set of edges.
6. The method of claim 1 ,
wherein the sequence includes a first video frame and a second video frame,
wherein the set of edges comprises first edges and second edges,
wherein detecting the set comprises detecting the first edges in the first video frame and detecting the second edges in the second video frame, and
wherein identifying the subset comprises (i) applying a logical conjunction on the detected first and second edges to produce an output that indicates common edges detected in both the first video frame and the second video frame, and (ii) deeming the common edges to be the edges of the subset.
7. The method of claim 6 ,
wherein the edge-detection process generates (i) a first binary representation of the first edges detected in the first video frame and (ii) a second binary representation of the second edges detected in the second video frame,
wherein applying the logical conjunction on the detected first and second edges comprises applying the logical conjunction on the first and second binary representations, and
wherein the output comprises a third binary representation of the common edges.
8. The method of claim 1 , wherein detecting, based on the edges of the identified subset, the video frame region that depicts the scoreboard comprises:
based on the edges of the identified subset, engaging in a filtering process to determine actual edges of the scoreboard image elements, wherein the filtering process comprises (i) recognizing edges, from among the edges of the identified subset, based on the recognized edges having a predetermined characteristic and (ii) deeming the recognized edges to be the actual edges of the scoreboard image elements; and
detecting, based on the determined actual edges of the scoreboard image elements, the video frame region that depicts the scoreboard.
9. The method of claim 8 , wherein recognizing edges based on the recognized edges having the predetermined characteristic comprises recognizing edges that have respective intensity values that are higher than a threshold intensity value.
10. The method of claim 8 , wherein detecting, based on the determined actual edges of the scoreboard image elements, the video frame region that depicts the scoreboard comprises:
engaging in a morphological-dilatation process to identify a substantially uniform region based on the determined actual edges of the scoreboard image elements; and
based on the substantially uniform region, determining the video frame region that depicts the scoreboard.
11. The method of claim 1 , wherein detecting, based on the edges of the identified subset, the video frame region that depicts the scoreboard comprises:
engaging in a morphological-dilatation process to identify a substantially uniform region based on the edges of the identified subset; and
based on the substantially uniform region, determining the video frame region that depicts the scoreboard.
12. The method of claim 1 ,
wherein the computing system has access to region-information indicating a predefined frame region that is not a candidate for depicting the scoreboard,
wherein engaging in the edge-detection process comprises, based on the region-information, excluding, from the edge-detection process, detection of edges in the predefined frame region of the video frames, and
wherein detecting the set of edges comprises detecting the set of edges within one or more frame regions other than the predefined frame region.
13. The method of claim 12 , where the predefined frame region is a center frame region of the video frames.
14. The method of claim 1 , further comprising:
carrying out, by the computing system, an action based on the detecting of the video frame region that depicts the scoreboard.
15. The method of claim 14 , wherein carrying out the action comprises one or more of the following:
(i) recording the detected video frame region,
(ii) outputting a report related to the detected video frame region,
(iii) determining accuracy of the detected video frame region, and
(iv) executing a machine-driven pattern recognition procedure that analyzes the detected video frame region to detect, within the detected video frame region, one or more other video frame regions based on the one or more other video frame regions depicting text of the scoreboard.
16. The method of claim 1 ,
wherein the first action comprises at least one action selected from the group consisting of (i) establishing a record of the detected scoreboard frame region and (ii) engaging in a further procedure to detect in the scoreboard frame region one or more sub-regions that depict text of the scoreboard, and
wherein the second action that corresponds to the detected video frame region being inaccurate comprises at least one action selected from the group consisting of (i) repeating the receiving, engaging, identifying, and detecting, and (ii) outputting an error result.
17. A computing system comprising:
a processing unit;
non-transitory data storage; and
program instructions stored in the non-transitory data storage and executable by the processing unit to carry out operations for automatically detecting, in a sequence of video frames, a video frame region that depicts a scoreboard, the operations comprising:
receiving the sequence of video frames, wherein the video frames of the sequence depict image elements including (i) scoreboard image elements that are unchanging across the video frames of the sequence and (ii) other image elements that change across the video frames of the sequence,
engaging in an edge-detection process to detect, in the video frames of the sequence, a set of edges of the depicted image elements,
identifying a subset of the detected set of edges based on each edge of the subset being unchanging across the video frames of the sequence, wherein the edges of the identified subset are candidate edges of the scoreboard image elements,
detecting, based on the edges of the identified subset, the video frame region that depicts the scoreboard,
executing a machine-driven pattern recognition procedure to extract a sport score from the detected video frame region in a given video frame of the sequence,
making a determination of whether the machine-driven pattern recognition procedure successfully extracted the sport score from the detected video frame region in the given video frame,
if the determination is that the machine-driven pattern recognition procedure successfully extracted the sport score from the detected video frame region in the given video frame, then, responsive to making the determination, taking a first action that corresponds to the detected video frame region being accurate, and
if the determination is that the machine-driven pattern recognition procedure did not successfully extract the sport score from the detected video frame region in the given video frame, then, responsive to making the determination, taking a second action that corresponds to the detected video frame region being inaccurate.
18. The computing system of claim 17 , the operations further comprising:
carrying out an action based on the detecting of the video frame region that depicts the scoreboard, wherein carrying out the action comprises one or more of the following:
recording the detected video frame region,
(ii) outputting a report related to the detected video frame region,
(iii) determining accuracy of the detected video frame region, and
(iv) executing a machine-driven pattern recognition procedure that analyzes the detected video frame region to detect, within the detected frame region, one or more other video frame regions based on the one or more other video frame regions depicting text of the scoreboard.
19. The computing system of claim 17 ,
wherein the first action comprises at least one action selected from the group consisting of (i) establishing a record of the detected scoreboard frame region and (ii) engaging in a further procedure to detect in the scoreboard frame region one or more sub-regions that depict text of the scoreboard, and
wherein the second action that corresponds to the detected video frame region being inaccurate comprises at least one action selected from the group consisting of (i) repeating the receiving, engaging, identifying, and detecting, and (ii) outputting an error result.
20. A non-transitory computer readable medium having stored thereon instructions executable by a processing unit to cause a computing system to perform operations for automatically detecting, in a sequence of video frames, a video frame region that depicts a scoreboard, the operations comprising:
receiving the sequence of video frames, wherein the video frames of the sequence depict image elements including (i) scoreboard image elements that are unchanging across the video frames of the sequence and (ii) other image elements that change across the video frames of the sequence;
engaging in an edge-detection process to detect, in the video frames of the sequence, a set of edges of the depicted image elements;
identifying a subset of the detected set of edges based on each edge of the subset being unchanging across the video frames of the sequence, wherein the edges of the identified subset are candidate edges of the scoreboard image elements;
detecting, based on the edges of the identified subset, the video frame region that depicts the scoreboard;
executing a machine-driven pattern recognition procedure to extract a sport score from the detected video frame region in a given video frame of the sequence;
making a determination of whether the machine-driven pattern recognition procedure successfully extracted the sport score from the detected video frame region in the given video frame;
if the determination is that the machine-driven pattern recognition procedure successfully extracted the sport score from the detected video frame region in the given video frame, then, responsive to making the determination, taking a first action that corresponds to the detected video frame region being accurate; and
if the determination is that the machine-driven pattern recognition procedure did not successfully extract the sport score from the detected video frame region in the given video frame, then, responsive to making the determination, taking a second action that corresponds to the detected video frame region being inaccurate.
21. The non-transitory computer readable medium of claim 20 ,
wherein the first action comprises at least one action selected from the group consisting of (i) establishing a record of the detected scoreboard frame region and (ii) engaging in a further procedure to detect in the scoreboard frame region one or more sub-regions that depict text of the scoreboard, and
wherein the second action that corresponds to the detected video frame region being inaccurate comprises at least one action selected from the group consisting of (i) repeating the receiving, engaging, identifying, and detecting, and (ii) outputting an error result.