Video processing for embedded information card localization and content extraction
Metadata for one or more highlights of a video stream may be extracted from one or more card images embedded in the video stream. The highlights may be segments of the video stream, such as a broadcast of a sporting event, that are of particular interest. According to one method, video frames of the video stream are stored. One or more information cards embedded in a decoded video frame may be detected by analyzing one or more predetermined video frame regions. Image segmentation, edge detection, and/or closed contour identification may then be performed on identified video frame region(s). Further processing may include obtaining a minimum rectangular perimeter area enclosing all remaining segments, which may then be further processed to determine precise boundaries of information card(s). The card image(s) may be analyzed to obtain metadata, which may be stored in association with at least one of the video frames.
1. A method for extracting metadata from a video stream, the method comprising:
at a data store, storing video frames of the video stream;
at a processor, automatically identifying and extracting a card image embedded in at least one video frame of the video frames by:
determining whether a predetermined location of the card image is known based on a network associated with the video stream;
based on determining that the predetermined location is known, processing the predetermined location, within the video frame, that defines a video frame region containing the card image; and
based on determining that the predetermined location is not known, sequentially processing a plurality of regions of the video frame to identify the video frame region containing the card image;
at the processor, analyzing the card image to obtain the metadata; and
at the data store, storing the metadata in association with at least one of the video frames.
2. The method of claim 1 , wherein:
the video stream comprises a broadcast of a sporting event;
the video frames constitute a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of a status of the sporting event during the highlight.
3. The method of claim 2 , further comprising, at an output device, presenting the metadata during viewing of the highlight.
4. The method of claim 3 , wherein automatically identifying and extracting the card image and analyzing the card image to obtain the metadata are carried out, for the highlight, during the viewing of the highlight.
5. The method of claim 1 , further comprising localizing and extracting the card image from the video frame region.
6. The method of claim 5 , wherein the localizing and extracting the card image from the video frame region comprises cropping the video frame to isolate the video frame region.
7. The method of claim 5 , wherein the localizing and extracting the card image from the video frame region comprises:
segmenting the video frame region, or a processed version of the video frame region, to generate a segmented image; and
modifying pixel values of segments adjacent to boundaries of the segmented image.
8. The method of claim 5 , wherein the localizing and extracting the card image from the video frame region comprises removing background from the video frame region or a processed version of the video frame region.
9. The method of claim 5 , wherein the localizing and extracting the card image from the video frame region comprises:
generating an edge image based on the video frame region with removed background;
finding contours in the edge image;
approximating the contours as polygons; and
extracting a region enclosed by a minimum rectangular perimeter encompassing all of the contours to generate a perimeter rectangular image.
10. The method of claim 9 , further comprising iteratively:
counting color-modified pixels for each edge of the perimeter rectangular image; and
moving any boundary edge, with a number of color-modified pixels exceeding a threshold, inward.
11. The method of claim 10 , further comprising validating a quadrilateral detected within the region by:
counting:
a first number of pixels in the video frame region;
a second number of pixels in the perimeter rectangular image; and
a third number of pixels in an adjusted perimeter rectangular image; and
comparing the first number, the second number, and the third number to determine whether an assumed quadrilateral within the region is viable.
12. The method of claim 5 , wherein the localizing and extracting the card image from the video frame region comprises adjusting a left boundary of the card image.
13. A non-transitory computer-readable medium for extracting metadata from a video stream, comprising instructions stored thereon, that when executed by a processor, perform the steps of:
causing a data store to store video frames of the video stream;
automatically identifying and extracting a card image embedded in at least one video frame of the video frames by performing at least one of:
determining whether a predetermined location of the card image is known based on a network associated with the video stream;
based on determining that the predetermined location is known, processing the predetermined location, within the video frame, that defines a video frame region containing the card image; and
based on determining that the predetermined location is not known, sequentially processing a plurality of regions of the video frame to identify the video frame region containing the card image;
analyzing the card image to obtain the metadata; and
causing the data store to store the metadata in association with at least one of the video frames.
14. The non-transitory computer-readable medium of claim 13 , wherein:
the video stream comprises a broadcast of a sporting event;
the video frames constitute a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of a status of the sporting event during the highlight.
15. The non-transitory computer-readable medium of claim 14 , further comprising instructions stored thereon, that when executed by a processor, cause an output device to present the metadata during viewing of the highlight.
16. The non-transitory computer-readable medium of claim 15 , wherein automatically identifying and extracting the card image and analyzing the card image to obtain the metadata are carried out, for the highlight, during the viewing of the highlight.
17. The non-transitory computer-readable medium of claim 13 , further comprising instructions stored thereon, that when executed by a processor, localize and extract the card image from the video frame region.
18. The non-transitory computer-readable medium of claim 17 , wherein the localizing and extracting the card image from the video frame region comprises cropping the video frame to isolate the video frame region.
19. The non-transitory computer-readable medium of claim 17 , wherein the localizing and extracting the card image from the video frame region comprises:
segmenting the video frame region, or a processed version of the video frame region, to generate a segmented image; and
modifying pixel values of segments adjacent to boundaries of the segmented image.
20. The non-transitory computer-readable medium of claim 17 , wherein the localizing and extracting the card image from the video frame region comprises removing background from the video frame region or a processed version of the video frame region.
21. The non-transitory computer-readable medium of claim 17 , wherein the localizing and extracting the card image from the video frame region comprises:
generating an edge image based on the video frame region with removed background;
finding contours in the edge image;
approximating the contours as polygons; and
extracting a region enclosed by a minimum rectangular perimeter encompassing all of the contours to generate a perimeter rectangular image.
22. A system for extracting metadata from a video stream, the system comprising:
a data store configured to store video frames of the video stream; and
a processor configured to:
automatically identify and extract a card image embedded in at least one video frame of the video frames by performing at least one of:
determining whether a predetermined location of the card image is known based on a network associated with the video stream;
based on determining that the predetermined location is known, processing the predetermined location, within the video frame, that defines a video frame region containing the card image; and
based on determining that the predetermined location is not known, sequentially processing a plurality of regions of the video frame to identify the video frame region containing the card image; and
analyze the card image to obtain the metadata;
wherein the data store is further configured to store the metadata in association with at least one of the video frames.
23. The system of claim 22 , wherein:
the video stream comprises a broadcast of a sporting event;
the video frames constitute a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of a status of the sporting event during the highlight.
24. The system of claim 23 , further comprising an output device configured to present the metadata during viewing of the highlight.
25. The system of claim 24 , wherein the processor is further configured to automatically identify and extract the card image and analyze the card image to obtain the metadata for the highlight during the viewing of the highlight.
26. The system of claim 22 , wherein the processor is further configured to localize and extract the card image from the video frame region.
27. The system of claim 26 , wherein the processor is further configured to localize and extract the card image from the video frame region by cropping the video frame to isolate the video frame region.
28. The system of claim 26 , wherein the processor is further configured to localize and extract the card image from the video frame region by:
segmenting the video frame region, or a processed version of the video frame region, to generate a segmented image; and
modifying pixel values of segments adjacent to boundaries of the segmented image.
29. The system of claim 26 , wherein the processor is further configured to localize and extract the card image from the video frame region by removing background from the video frame region or a processed version of the video frame region.
30. The system of claim 26 , wherein the processor is further configured to localize and extract the card image from the video frame region by:
generating an edge image based on the video frame region with removed background;
finding contours in the edge image;
approximating the contours as polygons; and
extracting a region enclosed by a minimum rectangular perimeter encompassing all of the contours to generate a perimeter rectangular image.