Methods and systems of combining video content with one or more augmentations
Data processing systems and methods are disclosed for combining video content with one or more augmentations to produce augmented video. Objects within video content may have associated bounding boxes that may each be associated with respective RGB values. Upon user selection of a pixel, the RGBA value of the pixel may be used to determine a bounding box associated with the RGBA value. The client may transmit an indicator of the determined bounding box to an augmentation system to request augmentation data for the object associated with the bounding box. The system then uses the indicator to determine the augmentation data and transmits the augmentation data to the client device.
1. A computer-implemented data processing method for displaying augmented content on a client device, the method comprising:
receiving, from an external server by one or more processors, video data corresponding to an event, the video data comprising video content and a plurality of definitions of a plurality of bounding boxes;
presenting, by one or more processors on a graphical user interface, the video content;
detecting, by one or more processors, a user selection of a portion of the graphical user interface;
determining, by one or more processors, a red, green, blue, alpha (RGBA) value associated with the user selection of the portion of the graphical user interface;
determining, by one or more processors, a bounding box RGBA value that corresponds to the RGBA value associated with the user selection of a portion of the graphical user interface, wherein the bounding box RGBA value is associated with a particular bounding box from among the plurality of bounding boxes;
transmitting, by one or more processors, an indication of the particular bounding box to a renderer;
receiving, by one or more processors from the renderer, augmentation data associated with the bounding box associated with the portion of the graphical user interface selected by the user;
generating, by one or more processors, augmented video content based on the video data and the augmentation data associated with the bounding box associated with the portion of the graphical user interface selected by the user; and
presenting, by one or more processors on the graphical user interface, the augmented video content.
2. The computer-implemented data processing method of claim 1 , wherein each definition of the plurality of definitions of bounding boxes defines an alpha value of 0 for a respective bounding box.
3. The computer-implemented data processing method of claim 1 , wherein each definition of the plurality of definitions of bounding boxes is associated with a respective object represented in the video content.
4. The computer-implemented data processing method of claim 3 , wherein the augmentation data associated with the bounding box associated with the portion of the graphical user interface selected by the user comprises an augmentation image and video frame data and location data associated with the augmentation image.
5. The computer-implemented data processing method of claim 4 , wherein transmitting the indication of the particular bounding box to the renderer causes the renderer to select the augmentation image from among a plurality of augmentation images associated with the respective object.
6. The computer-implemented data processing method of claim 1 , wherein the renderer comprises one of an external rendering engine and a client-side rendering engine.
7. The computer-implemented data processing method of claim 1 , wherein detecting the user selection of the portion of the graphical user interface comprises detecting a user tap of a display on the client device displaying the graphical user interface.
8. A video content augmentation system configured on a client device, the video content augmentation system comprising:
one or more computer processors;
memory storing computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
receiving, from an external server, video data corresponding to an event, the video data comprising video content and a plurality of transparent bounding boxes;
presenting, on a graphical user interface, the video content;
detecting a user selection of a portion of the graphical user interface;
determining a red, green, blue, alpha (RGBA) value associated with the user selection of the portion of the graphical user interface;
selecting a particular transparent bounding box from among the plurality of transparent bounding boxes by determining that an RGBA value associated with the particular transparent bounding box corresponds to the RGBA value associated with the user selection of the portion of the graphical user interface;
transmitting an indicator of the particular transparent bounding box to a renderer;
receiving, from the renderer, one or more augmentation images associated with the particular transparent bounding box;
receiving, from the renderer, video frame data and location data associated with the one or more augmentation images; and
presenting the one or more augmentation images on the graphical user interface based on the video frame data and the location data.
9. The video content augmentation system of claim 8 , wherein each of the one or more augmentation images is a Portable Network Graphics (PNG) image.
10. The video content augmentation system of claim 8 , wherein the video data further comprises a respective pre-defined RGBA value for each bounding box of the plurality of transparent bounding boxes.
11. The video content augmentation system of claim 8 , wherein each bounding box of the plurality of transparent bounding boxes is associated with a respective object represented in the video content.
12. The video content augmentation system of claim 11 , wherein the event is a sporting event, and
wherein a respective object associated with at least one bounding box of the plurality of transparent bounding boxes corresponds to a player in the sporting event.
13. The video content augmentation system of claim 11 , wherein the event is a sporting event, and
wherein a respective object associated with at least one bounding box of the plurality of transparent bounding boxes corresponds to a non-player object in the sporting event.
14. The video content augmentation system of claim 8 , wherein presenting the one or more augmentation images on the graphical user interface based on the video frame data and the location data comprises presenting the one or more augmentation images on the graphical user interface in conjunction with the video content so that, when the one or more augmentation images are presented, the one or more augmentation images remain in a substantially fixed orientation relative to an object associated with the one or more augmentation images as the video content is presented on the graphical user interface.
15. A non-transitory computer-readable medium storing computer-executable instructions for:
determining a plurality of objects within each frame of a plurality of video frames of video content;
assigning a plurality of bounding boxes to the plurality of objects, wherein each bounding box is associated with a respective object represented in one or more frames of the plurality of video frames;
assigning a respective red, green, blue, alpha (RGBA) value to each bounding box of the plurality of bounding boxes;
transmitting, to a client device, video data corresponding to an event, the video data comprising the video content, the plurality of bounding boxes, and the assigned RGBA values;
receiving, from the client device, an indicator of a particular bounding box identified from among the plurality of bounding boxes, wherein the particular bounding box is identified based on the assigned RGBA values of the plurality of bounding boxes;
determining a particular object within a frame of the plurality of video frames of the video content based on the indicator of the particular bounding box;
determining a current augmentation state for the particular object;
selecting an augmentation image from among a plurality of augmentation images associated with the particular object based on the current augmentation state for the particular object;
determining video frame data and location data associated with the particular object; and transmitting the augmentation image, the video frame data, and the location data to the client device.
16. The non-transitory computer-readable medium of claim 15 , further comprising computer-executable instructions for:
assigning an alpha value of 0 to each bounding box of the plurality of bounding boxes.
17. The non-transitory computer-readable medium of claim 15 , wherein determining a current augmentation state for the particular object comprises determining that there is no current augmentation image associated with the particular object.
18. The non-transitory computer-readable medium of claim 15 , wherein selecting the augmentation image from among the plurality of augmentation images associated with the particular object based on the current augmentation state for the particular object comprises:
selecting a next sequential augmentation image from among a sequence of augmentation images, wherein the current augmentation state is associated with a current augmentation image, and wherein the next sequential augmentation image follows the current augmentation image in the sequence of augmentation images.
19. The non-transitory computer-readable medium of claim 15 , wherein determining a current augmentation state for the particular object comprises determining that there is no current augmentation image associated with the particular object.