Methods and systems of combining video content with one or more augmentations to produce augmented video
Data processing systems and methods are disclosed for combining video content with one or more augmentations to produce augmented video. Objects within video content may have associated bounding boxes that may each be associated with respective RGB values. Upon user selection of a pixel, the RGBA value of the pixel may be used to determine a bounding box associated with the RGBA value. The client may transmit an indicator of the determined bounding box to an augmentation system to request augmentation data for the object associated with the bounding box. The system then uses the indicator to determine the augmentation data and transmits the augmentation data to the client device.
1. A computer-implemented data processing method for displaying augmented content on a client device, the method comprising:
receiving, from an external server by the client device, video data corresponding to an event, the video data comprising video content and a plurality of definitions of a plurality of bounding areas;
presenting, on a graphical user interface displayed via the client device, the video content;
detecting, by the client device, a user selection of a portion of the graphical user interface;
determining, by the client device, a value associated with one or more pixels associated with the portion of the graphical user interface;
determining, by the client device, a particular bounding area from the plurality of bounding areas based at least in part on a bounding area value associated with the particular bounding area that corresponds to the value associated with the one or more pixels associated with the portion of the graphical user interface;
transmitting, by the client device, an indication of the particular bounding area to a renderer;
receiving, by the client device from the renderer, augmentation data associated with the particular bounding area;
generating, by the client device, augmented video content based on the video data and the augmentation data associated with the particular bounding area; and
presenting, by the client device on the graphical user interface, the augmented video content.
2. The computer-implemented data processing method of claim 1 , wherein each definition of the plurality of definitions is associated with a respective object represented in the video content.
3. The computer-implemented data processing method of claim 2 , wherein the augmentation data associated with the particular bounding area comprises an augmentation image and video frame data and auxiliary data associated with the augmentation image.
4. The computer-implemented data processing method of claim 3 , wherein transmitting the indication of the particular bounding area to the renderer causes the renderer to select the augmentation image from among a plurality of augmentation images associated with the respective object.
5. The computer-implemented data processing method of claim 1 , wherein the renderer comprises one of an external rendering engine and a client-side rendering engine.
6. The computer-implemented data processing method of claim 1 , wherein detecting the user selection of the portion of the graphical user interface comprises detecting a user tap or a user click of a display on the client device displaying the graphical user interface.
7. A client device, the client device comprising:
one or more computer processors;
memory storing computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
receiving, from an external server, video data corresponding to an event, the video data comprising video content and a plurality of transparent bounding areas;
presenting, on a graphical user interface, the video content;
detecting a user selection of a portion of the graphical user interface;
determining a color value associated with the portion of the graphical user interface;
selecting a particular transparent bounding area from among the plurality of transparent bounding areas by determining that a transparent bounding area color value associated with the particular transparent bounding area corresponds to the color value associated with the portion of the graphical user interface;
transmitting an indicator of the particular transparent bounding area to a renderer;
receiving, from the renderer, one or more augmentation images associated with the particular transparent bounding area;
receiving, from the renderer, video frame data and location data associated with the one or more augmentation images; and
presenting the one or more augmentation images on the graphical user interface based on the video frame data and the location data.
8. The client device of claim 7 , wherein each of the one or more augmentation images is a Portable Network Graphics (PNG) image.
9. The client device of claim 7 , wherein the video data further comprises a respective pre-defined transparent bounding area color value for each transparent bounding area of the plurality of transparent bounding areas.
10. The client device of claim 7 , wherein each transparent bounding area of the plurality of transparent bounding areas is associated with a respective object represented in the video content.
11. The client device of claim 10 , wherein the event is a sporting event, and
the respective object associated with at least one transparent bounding area of the plurality of transparent bounding areas corresponds to a player in the sporting event.
12. The client device of claim 10 , wherein the event is a sporting event, and
the respective object associated with at least one transparent bounding area of the plurality of transparent bounding areas corresponds to a non-player object in the sporting event.
13. The client device of claim 7 , wherein presenting the one or more augmentation images on the graphical user interface based on the video frame data and the location data comprises presenting the one or more augmentation images on the graphical user interface in conjunction with the video content so that, when the one or more augmentation images are presented, the one or more augmentation images remain in a substantially fixed orientation relative to an object associated with the one or more augmentation images as the video content is presented on the graphical user interface.
14. The client device of claim 7 , wherein the color value associated with the portion of the graphical user interface comprises a red, green, blue, alpha (RGBA) value.
15. A non-transitory computer-readable medium storing computer-executable instructions, the computer-executable instructions executable by one or more processors for performing operations comprising
determining a plurality of objects within a plurality of video frames of video content;
assigning a plurality of bounding areas to the plurality of objects, wherein each bounding area is associated with a respective object represented in one or more frames of the plurality of video frames;
assigning a respective color value to each bounding area of the plurality of bounding areas;
transmitting, to a client device, video data corresponding to an event, wherein the video data comprises the video content, the plurality of bounding areas, and the assigned color values;
receiving, from the client device, an indicator of a particular bounding area identified from among the plurality of bounding areas, wherein the particular bounding area is identified based on the respective color value assigned to the particular bounding area;
determining a particular object within a frame of the plurality of video frames of the video content based on the indicator of the particular bounding area;
determining a current augmentation state for the particular object;
selecting an augmentation image from among a plurality of augmentation images associated with the particular object based on the current augmentation state for the particular object;
determining video frame data and location data associated with the particular object; and
transmitting the augmentation image, the video frame data, and the location data to the client device.
16. The non-transitory computer-readable medium of claim 15 , wherein each of the respective color values assigned to each bounding area of the plurality of bounding areas comprises an opacity value.
17. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
assigning an opacity value of zero to each bounding area of the plurality of bounding areas.
18. The non-transitory computer-readable medium of claim 15 , wherein determining a current augmentation state for the particular object comprises determining that there is no current augmentation image associated with the particular object.
19. The non-transitory computer-readable medium of claim 15 , wherein selecting the augmentation image from among the plurality of augmentation images associated with the particular object based on the current augmentation state for the particular object comprises:
selecting a next sequential augmentation image from among a sequence of augmentation images, wherein the current augmentation state is associated with a current augmentation image, and the next sequential augmentation image follows the current augmentation image in the sequence of augmentation images.
20. The non-transitory computer-readable medium of claim 15 , wherein each of the respective color values assigned to each bounding area of the plurality of bounding areas comprises a red, green, blue, alpha (RGBA) value.