Producing and adapting video images for presentation on displays with different aspect ratios
The described embodiments include systems and methods for producing and adapting images, such as video images, for presentation on display devices that have various different aspects ratios, such as 4:3, 16:9, 9:16, etc. In one embodiment, a method for producing content, such as video images, can begin by selecting an original aspect ratio and determining, within at least a first scene in the content, a position of a subject in the first scene. In one embodiment, the original aspect ratio can be substantially square (e.g., 1:1). Metadata can then be created, based on the position of the subject in the first scene, to guide playback devices to asymmetrically crop the content, relative to the position, for display on display devices that have aspect ratios that are different than the original aspect ratio. Other methods and systems are also described.
1. A machine implemented method, the method comprising:
receiving content that comprises image data for at least a first scene and receiving first metadata that is associated with the first scene, the first metadata specifying how, relative to a first position of a first subject in the first scene, to adapt a playback on a display device having an aspect ratio (AR) that is different than an original aspect ratio, the first scene having been created on an image canvas that has the original aspect ratio; and
adapting output to the aspect ratio of the display device based on the first metadata; further comprising:
receiving distance and position parameters related to a position of a viewer relative to the display device; and
further adapting the output of the first subject to the display device based on the distance and position parameters; and
wherein the content includes a plurality of scenes, including the first scene, and each of the scenes in the plurality of scenes has a determined position for a subject of the corresponding scene, wherein subjects are determined on a scene by scene basis, and wherein adapting for different ARs is done on a scene basis and wherein tone mapping is done on a scene by scene or frame by frame basis for the display device based on what relative portion of an adapted image is labeled as a region of interest, wherein the region of interest includes the first subject, and wherein each scene includes one or more frames.
2. The method as in claim 1 wherein the original AR is substantially square.
3. The method as in claim 2 wherein substantially square is more square than a 16:9 AR such that the ratio of the length to height for the original AR is less than the ratio of 16:9 (16/9) and wherein the original AR varies during the content.
4. The method as in claim 1 , wherein the first metadata guides asymmetric cropping on a playback device to expand from the first subject in the first scene for different ARs when adapting for playback.
5. The method as in claim 1 , wherein further adapting the output of the first subject to the display device comprises up-scaling the output of the first subject when a viewing distance between the viewer and the display device increases and down-scaling the output of the first subject when the viewing distance between the viewer and the display device decreases.
6. The method as in claim 1 , wherein further adapting the output of the first subject to the display device comprises shifting the output of the first subject to the left when the display device moves to the right relatively to the viewer and shifting the output of the first subject to the right when the display device moves to the left relatively to the viewer.
7. The method of claim 1 , further comprising:
receiving graphics data; and
generating a video output comprising a composite of the graphics data and the adapted output.
8. The method of claim 1 , wherein the first metadata further comprise syntax elements for defining an intended viewer motion path to guide Ken Burns-related effects during playback.
9. A non-transitory machine-readable medium storing executable program instructions which when executed by a data processing system cause the data processing system to perform a method as in claim 1 .
10. A data processing system having a processing system and memory, the data processing system configured to perform a method as in claim 1 .
11. A machine implemented method, the method comprising:
selecting an original aspect ratio (AR) for an image canvas for use in content creation;
determining, within at least a first scene in content on the image canvas, a first position of a first subject in the at least first scene;
determining, based on a determined position of the first subject and based on a distance between a viewer and a display device, first metadata that specifies how, relative to the first position, to adapt a playback on the display device having an AR that is different than the original AR; and wherein the content includes a plurality of scenes, including the first scene, and each of the scenes in the plurality of scenes has a determined position for a subject of the corresponding scene, wherein subjects are determined on a scene by scene basis, and wherein adapting for different ARs is adapted to be done on a scene basis and wherein tone mapping is adapted to be done on a scene by scene or frame by frame basis for the display device based on what relative portion of an adapted image is labeled as a region of interest, wherein the region of interest includes the first subject, and wherein each scene includes one or more frames; and
storing the first metadata, wherein the first metadata and the content are to be used, or are to be transmitted for use, during playback.
12. The method as in claim 11 , wherein different zoom factors for displaying the first subject are provided for different distances between the viewer and the display device.
13. The method as in claim 11 , wherein the original AR is substantially square.
14. The method as in claim 13 wherein substantially square is one of either (1) more square than a 16:9 AR such that the ratio of the length to height for the original AR is less than the ratio of 16:9 (16/9) but greater than or equal to 1:1 or (2) more than the ratio of 9:16 but less than 1:1 when a portrait mode is preferred; and wherein the original AR varies during the content.
15. The method as in claim 11 , the method further comprising:
determining, for a plurality of scenes, a plurality of subjects, the plurality of scenes including the first scene and the plurality of subjects including the first subject; and
determining for each of the subjects in the plurality of scenes a corresponding position within the corresponding scene.
16. The method as in claim 15 , wherein subjects are determined on a scene by scene basis within the plurality of scenes; and wherein the method further comprises:
displaying a preview of how different aspects ratios will crop based on the metadata.
17. The method as in claim 11 wherein the first metadata guides asymmetric cropping on a playback device to expand from the first subject in the first scene for different ARs when adapting for playback.
18. A non-transitory machine-readable medium storing executable program instructions which when executed by a data processing system cause the data processing system to perform a method as in claim 11 .
19. A data processing system having a processing system and memory, the data processing system configured to perform a method as in claim 11 .