Production platform
Camera equipment used for capturing a live event is expensive and may require professionals to operate the equipment. Such camera equipment is not accessible for capturing events and experiences by everyday users. Suppose media content is captured by lower-cost input devices such as mobile phones with a video camera. A production platform can use computer vision techniques to produce enhanced media content from media content captured by the lower-cost input devices. In one example, the production platform can aggregate and analyze the received media content to extract information and determine blind spots. The production platform can create or retrieve content based on the extracted information and insert the content in the blind spots. The production platform can deliver enhanced media content to an end user device, the enhanced media content may have the received media content and the inserted content.
1 . A method comprising:
receiving a plurality of media content captured by a plurality of user devices at different moving viewpoints during a time period;
determining that two or more of the plurality of media content are capturing a same area from multiple viewpoints during an epoch of the time period;
creating an event model, wherein the event model includes a timeline of the time period and a map of a location of an event;
reconstructing the area for the event model spatially and temporally for the epoch using the two or more of the plurality of media content capturing the area, wherein reconstructing the area comprises extracting one or more four-dimensional assets, the one or more four-dimensional assets including three-dimensional spatial information and temporal information;
identifying, for the event model, a blind spot based on the three-dimensional spatial information or the temporal information having insufficient coverage spatially or temporally respectively, wherein the blind spot includes one or more of a spatial blind spot and a temporal blind spot;
transforming the reconstructed area into multimedia content, wherein transforming the reconstructed area comprises generating filler content for the blind spot based on the event model and inserting the filler content into the blind spot; and
delivering the multimedia content to an end user device.
2 . The method of claim 1 , wherein determining that the two or more of the plurality of media content are capturing the same area comprises determining that the two or more of the plurality of media content are capturing the same area based on location information and field of view information associated with the plurality of media content.
3 . The method of claim 1 , wherein determining that the two or more of the plurality of media content are capturing the same area comprises determining that the two or more of the plurality of media content are capturing the same area based on a common detection of one or more objects captured in the two or more of the plurality of media content.
4 . The method of claim 1 , wherein transforming the reconstructed area into the multimedia content comprises packaging the one or more four-dimensional assets into the multimedia content.
5 . The method of claim 1 , wherein transforming the reconstructed area into multimedia content comprises:
rendering the one or more four-dimensional assets according to a user-selected theme; and
generating the multimedia content from the rendering.
6 . The method of claim 1 , wherein transforming the reconstructed area into the multimedia content comprises:
determining a viewpoint of the area that has sufficient information about the area;
rendering the one or more four-dimensional assets using volume rendering based on the viewpoint; and
generating the multimedia content from the rendering.
7 . The method of claim 1 , wherein:
the spatial blind spot is an area that is not captured by the plurality of media content; and
the temporal blind spot is a further epoch of the time period that is not captured by the plurality of media content.
8 . The method of claim 1 , wherein generating the filler content for the blind spot based on the event model comprises:
retrieving content associated with the event model, the content including one or more images of the location of the event associated with the event model.
9 . The method of claim 1 , further comprising:
determining one or more contextual cues for the event model adjacent to the spatial blind spot or the temporal blind spot;
wherein generating the filler content for the blind spot comprises creating the filler content based on the one or more contextual cues.
10 . The method of claim 1 , wherein generating the filler content for the blind spot based on the event model comprises:
generating the filler content using one or more of a generative machine learning model and a computer vision technique.
11 . The method of claim 1 , wherein generating the filler content for the blind spot based on the event model comprises:
receiving one or more images of the location of the event; and
generating artificial content based on the one or more images of the location of the event.
12 . A production platform, comprising:
an aggregation and analysis part to:
receive a plurality of media content captured by a plurality of user devices at different moving viewpoints during a time period;
determine that two or more of the plurality of media content are capturing a same area from multiple viewpoints during an epoch of the time period; and
create an event model, wherein the event model includes a timeline of the time period and a map of a location of an event;
a content enhancement part to:
reconstruct the area for an event model spatially and temporally for the epoch using the two or more of the plurality of media content capturing the area by extracting one or more four-dimensional assets, the one or more four-dimensional assets including three-dimensional spatial information and temporal information;
identify, for the event model, a blind spot based on the three-dimensional spatial information or the temporal information having insufficient coverage spatially or temporally respectively, wherein the blind spot includes one or more of a spatial blind spot and a temporal blind spot; and
transform the reconstructed area into multimedia content by generating filler content for the blind spot based on the event model and inserting the filler content into the blind spot; and
a content delivery part to:
deliver the multimedia content to an end user device.
13 . The production platform of claim 12 , wherein determining that two or more of the plurality of media content are capturing the same area based on location information and field of view information associated with the plurality of media content.
14 . The production platform of claim 12 , wherein determining that two or more of the plurality of media content are capturing the same area based on a common detection of one or more objects captured in the two or more of the plurality of media content.
15 . The production platform of claim 12 , wherein transforming the reconstructed area into multimedia content comprises packaging the one or more four-dimensional assets into the multimedia content.
16 . The production platform of claim 12 , wherein transforming the reconstructed area into multimedia content comprises:
rendering the one or more four-dimensional assets according to a user-selected theme; and
generating the multimedia content from the rendering.
17 . The production platform of claim 12 , wherein transforming the reconstructed area into multimedia content comprises:
determining a viewpoint of the area that has sufficient information about the area;
rendering the one or more four-dimensional assets using volume rendering based on the viewpoint; and
generating the multimedia content from the rendering.
18 . One or more non-transitory computer-readable storage media comprising instructions for execution which, when executed by one or more processors, cause the one or more processors to:
process a plurality of media content captured by a plurality of user devices at different moving viewpoints during a time period;
determine that two or more of the plurality of media content are capturing a same area from multiple viewpoints during an epoch of the time period;
create an event model, wherein the event model includes a timeline of the time period and a map of a location of an event;
reconstruct the area for the event model spatially and temporally for the epoch using the two or more of the plurality of media content capturing the area, wherein reconstructing the area comprises extracting one or more four-dimensional assets, the one or more four-dimensional assets including three-dimensional spatial information and temporal information;
identify, for the event model, a blind spot based on the three-dimensional spatial information or the temporal information having insufficient coverage spatially or temporally respectively, wherein the blind spot includes one or more of a spatial blind spot and a temporal blind spot;
generate multimedia content based on the reconstructed area, wherein generating multimedia content comprises generating filler content for the blind spot based on the event model and inserting the filler content into the blind spot; and
transmit the multimedia content to an end user device.
19 . The one or more non-transitory computer-readable storage media of claim 18 , wherein determining that two or more of the plurality of media content are capturing the same area based on location information and field of view information associated with the plurality of media content.
20 . The one or more non-transitory computer-readable storage media of claim 18 , wherein determining that two or more of the plurality of media content are capturing the same area based on a common detection of one or more objects captured in the two or more of the plurality of media content.