Data processing systems and methods for enhanced augmentation of interactive video content
Data processing systems and methods are disclosed for augmenting video content with one or more augmentations to produce augmented video. Elements within video content may be identified by spatiotemporal indices and may have associated values. An advertiser can pay to have an augmentation added to an element that, for example, advertises the advertiser's goods and/or includes a link that, when activated, takes a user to the advertiser's web site. Elements may have associated contexts that can be used to determine augmentations and element value, such as a position and/or current use of the element.
1. A computer-implemented data processing method for generating augmented video content, the method comprising:
receiving, from an external server by one or more computer processors, video data corresponding to an event, the video data comprising video content and spatiotemporal data, the video content comprising a plurality of video frames;
determining, by one or more computer processors, based at least in part on the spatiotemporal data, one or more semantic elements in a video frame of the plurality of video frames;
determining, by one or more computer processors, based at least in part on the spatiotemporal data, one or more semantic contexts for each respective semantic element of the one or more semantic elements in the video frame of the plurality of video frames, wherein the one or more semantic contexts for each respective semantic element of the one or more semantic elements indicates an event associated with a respective semantic element of the one or more semantic elements;
determining, by one or more computer processors, based at least in part on the one or more semantic contexts and the one or more semantic elements, an augmentation for each respective semantic element of the one or more semantic elements in the video frame of the plurality of video frames;
determining, by one or more computer processors, based at least in part on each respective semantic element and the one or more semantic contexts for each respective semantic element of the one or more semantic elements, a presentation location within the video frame for the augmentation for each respective semantic element of the one or more semantic elements;
generating, by one or more computer processors, using the presentation location within the video frame for the augmentation for each respective semantic element of the one or more semantic elements, augmented video content comprising:
the video content; and
the augmentation for each respective semantic element of the one or more semantic elements configured at the presentation location within the video frame for the augmentation for each respective semantic element of the one or more semantic elements;
presenting, by one or more computer processors on a graphical user interface, the augmented video content;
detecting, by one or more computer processors, a user selection of a particular augmentation associated with a particular semantic element of the one or more semantic elements;
at least partially in response to detecting the user selection of the particular augmentation associated with a particular semantic element of the one or more semantic elements, determining one or more actions based at least in part on the particular semantic element of the one or more semantic elements; and
at least partially in response to determining the one or more actions, taking, by one or more computer processors, the one or more actions.
2. The computer-implemented data processing method of claim 1 , wherein:
the particular augmentation associated with the particular semantic element of the one or more semantic elements is associated with a link to a website; and
the one or more actions comprise directing a user computing device to the website.
3. The computer-implemented data processing method of claim 1 , wherein one or more of the one or more semantic elements is selected from a group consisting of:
(a) a person;
(b) an item worn by a person;
(c) a portion of an environment;
(d) an item in the environment; and
(c) a portion of an item in the environment.
4. The computer-implemented data processing method of claim 1 , wherein the spatiotemporal data comprises data indicating one or more regions of pixels, wherein each of the one or more regions of pixels corresponds to one or more pixels in the video frame of the plurality of video frames.
5. The computer-implemented data processing method of claim 4 , wherein determining the one or more semantic elements in the video frame of the plurality of video frames based at least in part on the spatiotemporal data comprises associating, by one or more computer processors, a particular region of pixels of the one or more regions of pixels with each of the one or more semantic elements.
6. The computer-implemented data processing method of claim 5 , wherein determining the one or more semantic contexts for each respective semantic element of the one or more semantic elements in the video frame of the plurality of video frames is further based at least in part on the particular region of pixels of the one or more regions of pixels associated with each respective semantic element of the one or more semantic elements.
7. The computer-implemented data processing method of claim 1 , wherein determining the augmentation for each respective semantic element of the one or more semantic elements in the video frame of the plurality of video frames comprises selecting the augmentation for each respective semantic element of the one or more semantic elements from one or more augmentations associated with each respective semantic element of the one or more semantic elements.
8. A video content augmentation system configured for generating augmented video content, the video content augmentation system comprising:
one or more computer processors;
memory storing computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
receiving, from an external server, video data corresponding to an event, the video data comprising a plurality of video frames and pixel data for each video frame of the plurality of video frames, wherein the pixels data comprises identification data for one or more regions of pixels in each video frame of the plurality of video frames;
identifying, based at least in part on the pixel data, a particular region of pixels of the one or more regions of pixels in a particular video frame of the plurality of video frames;
determining, based at least in part on the particular region of pixels, a particular semantic element in the particular video frame that is associated with the particular region of pixels;
determining, based at least in part on the pixel data, one or more semantic contexts for the particular semantic element, wherein the one or more semantic contexts for the particular semantic element indicates an event associated with the particular semantic element;
determining, based at least in part on the one or more semantic contexts and the particular semantic element, one or more augmentations for the particular semantic element;
determining, based at least in part on the one or more semantic contexts and the particular semantic element, a presentation location within the particular video frame for the one or more augmentations for the particular semantic element;
generating, using the presentation location within the particular video frame for the one or more augmentations for the particular semantic element, augmented video content comprising the particular video frame and the one or more augmentations for the particular semantic element configured at the respective presentation location within the video frame for each respective augmentation of the one or more augmentations for each respective semantic element of the one or more semantic elements;
transmitting the augmented video content to a user graphical display device;
receiving an indication of a user selection, on the user graphical display device, of a particular augmentation of the one or more augmentations for the particular semantic element;
at least partially in response receiving the indication of the user selection of the particular augmentation of the one or more augmentations for the particular semantic element, determining one or more actions based at least in part on the particular semantic element, the one or more semantic contexts for the particular semantic element, and the one or more augmentations for the particular semantic element; and
at least partially in response to determining the one or more actions, taking the one or more actions.
9. The video content augmentation system of claim 8 , wherein:
the particular augmentation of the one or more augmentations for the particular semantic element is an avatar associated with a second user; and
the one or more actions comprise generating a communications interface augmentation proximate to the avatar associated with the second user.
10. The video content augmentation system of claim 9 , wherein the operations further comprise presenting content received from the second user in the communications interface.
11. The video content augmentation system of claim 9 , wherein the operations further comprise receiving content from a user of the user graphical display device and presenting the received content in the communications interface.
12. The video content augmentation system of claim 8 , wherein the indication of the user selection of the particular augmentation of the one or more augmentations for the particular semantic element is generated at least partially in response to a user tap on the user graphical display device.
13. The video content augmentation system of claim 8 , wherein determining, based at least in part on the pixel data, the one or more semantic contexts for the particular semantic element comprises determining that the particular region of pixels corresponds to an area of the particular video frame that is unoccupied by any one or more persons.
14. The video content augmentation system of claim 8 , wherein the event is a sporting event.
15. A non-transitory computer-readable medium storing computer-executable instructions for generating augmented video content, the computer-executable instructions comprising instructions for:
receiving, from an external server by one or more computer processors, video data corresponding to an event, the video data comprising video content and spatiotemporal data, the video content comprising a plurality of video frames;
determining, by one or more computer processors, based at least in part on the spatiotemporal data, one or more semantic elements in a video frame of the plurality of video frames;
determining, by one or more computer processors, based at least in part on the spatiotemporal data, one or more semantic contexts for each respective semantic element of the one or more semantic elements in the video frame of the plurality of video frames, wherein the one or more semantic contexts for each respective semantic element of the one or more semantic elements indicates an event associated with a respective semantic element of the one or more semantic elements;
determining, by one or more computer processors, based at least in part on the one or more semantic contexts and the one or more semantic elements, an augmentation for each respective semantic element of the one or more semantic elements in the video frame of the plurality of video frames;
determining, by one or more computer processors, based at least in part on the one or more semantic contexts and the one or more semantic elements, a presentation location within the video frame of the plurality of video frames for the augmentation for each respective semantic element of the one or more semantic elements;
generating, by one or more computer processors, using the presentation location within the video frame of the plurality of video frames for the augmentation for each respective semantic element of the one or more semantic elements, augmented video content comprising:
the video content; and
the augmentation for each respective semantic element of the one or more semantic elements configured at the presentation location within the video frame of the plurality of video frames for the augmentation for each respective semantic element of the one or more semantic elements;
presenting, by one or more computer processors on a graphical user interface, the augmented video content;
detecting, by one or more computer processors, a user selection of a particular augmentation associated with a particular semantic element of the one or more semantic elements;
at least partially in response to detecting the user selection of the particular augmentation associated with a particular semantic element of the one or more semantic elements, determining a second particular augmentation associated with the particular semantic element of the one or more semantic elements based at least in part on the particular semantic element, the user selection of the particular augmentation associated with a particular semantic element, and a particular semantic context of the one or more semantic contexts associated with the particular semantic element;
generating, by one or more computer processors, second augmented video content comprising the video content and the second particular augmentation; and
presenting, by one or more computer processors on the graphical user interface, the second augmented video content.
16. The non-transitory computer-readable medium of claim 15 , wherein the second particular augmentation comprises player statistics associated with a player associated with the particular semantic element.
17. The non-transitory computer-readable medium of claim 15 , wherein the computer-executable instructions further comprise instructions for determining, by one or more computer processors, based at least in part on the particular semantic element and the particular semantic context of the one or more semantic contexts associated with the particular semantic element, a value for the particular augmentation associated with the particular semantic element.
18. The non-transitory computer-readable medium of claim 15 , wherein the particular augmentation associated with the particular semantic element comprising advertising content.
19. The non-transitory computer-readable medium of claim 15 , wherein determining the augmentation for each respective semantic element of the one or more semantic elements in the video frame of the plurality of video frames is further based at least in part on user context.
20. The non-transitory computer-readable medium of claim 19 , wherein the user context comprises one or more context items determined based at least in part on data selected from a group consisting of:
(a) user profile data;
(b) user interaction history data;
(c) user social media data;
(d) user online activity data; and
(e) user shopping history data.