Media rendering with orientation metadata
Systems and methods for cropping media for a particular orientation using a computing device are described. In some implementations, a method may first comprise receiving by a video preprocessor of the device, a first frame of media in a first orientation. A first region comprising a first feature within the first frame may be identified, by an image analyzer. A cropping calculator of the device may generate a score for the first region based on a characteristic of the first feature and determine that the score for the first region exceeds a threshold. An image processor of the device may then crop the first frame of the video, responsive to the determination that the score for the first region exceeds the threshold, to include the first region within a predetermined display area comprising a subset of the first frame in a second orientation.
1. A method, comprising:
receiving, by a mobile device, data of a first frame of a video;
displaying, on a display of the mobile device, the first frame of the video in a first display orientation;
receiving, by the mobile device, data of a second frame of the video;
detecting, by the mobile device, that the mobile device has been rotated to display the video in the second display orientation instead of the first display orientation; and
responsive to detecting that the mobile device has been rotated,
extracting, by a video preprocessor of the mobile device, from the data of the second frame, region data associated with a first region of the second frame of the video;
cropping, by the video preprocessor of the mobile device, the second frame of the video to a cropped size centered on the first region of the second frame and further based on an aspect ratio of the mobile device and a second display orientation to display the second frame of the video, the second display orientation is different from the first display orientation and is another of a portrait orientation or a landscape orientation; and
displaying, on the display of the mobile device, the cropped second frame of the video in the second display orientation.
2. The method of claim 1 , wherein the aspect ratio is a first aspect ratio, and wherein the method further comprises:
comparing, by the video preprocessor of the mobile device, a second aspect ratio of the video to the first aspect ratio of the mobile device positioned in the second display orientation to display the second frame of the video, the second aspect ratio different from the first aspect ratio; and
determining, by the video preprocessor of the mobile device, based on the first aspect ratio, to crop the second frame of the video to the cropped centered on the first region of the first frame and further based on the aspect ratio of the mobile device and the second display orientation to display the second frame of the video.
3. The method of claim 1 , wherein the region data is first region data, and wherein the method further comprises:
receiving, by the mobile device, data of a third frame of the video; and
responsive to receiving the data of the third frame of the video,
extracting, by the video preprocessor of the mobile device, from the data of the third frame, second region data associated with a second region of the third frame of the video, the second region is different from the first region of the second frame of the video;
cropping, by the video preprocessor of the mobile device, the third frame of the video to a cropped size centered on the second region of the third frame and further based on the aspect ratio of the mobile device and the second display orientation to display the third frame of the video; and
displaying, on the display of the mobile device, the cropped third frame of the video in the second display orientation.
4. The method of claim 1 , wherein the region data is first region data, and wherein the method further comprises:
receiving, by the mobile device, data of a third frame of the video;
detecting, by the mobile device, that the mobile device has been rotated to display the video in the first display orientation instead of the second display orientation; and
displaying, on the display of the mobile device, responsive to receiving the data of the third frame of the video, the third frame of the video in the first display orientation.
5. The method of claim 4 , wherein the aspect ratio is a first aspect ratio, and wherein the method further comprises:
comparing, by the video preprocessor of the mobile device, a second aspect ratio of the video to a third aspect ratio of the mobile device positioned in the first display orientation to display the third frame of the video; and
determining, by the video preprocessor of the mobile device, based on the second aspect ratio being less than the third aspect ratio, to display, on the display of the mobile device, the third frame of the video in the first display orientation.
6. The method of claim 1 , wherein the first frame of the video is displayed on the display of the mobile device responsive to the mobile device receiving the data of the first frame of the video.
7. The method of claim 1 , wherein responsive to detecting that the mobile device has been rotated to display the video in the second display orientation instead of the first display orientation, the method further comprises:
identifying, by the video preprocessor of the mobile device, from the data of the second frame, one or more objects within the second frame to center on;
determining, by the video preprocessor of the mobile device, based on the identified one or more objects within the second frame, the first region of the second frame of the video for extraction; and
wherein the extracted region data is associated with the first region determined based on the identified one or more objects.
8. The method of claim 7 , wherein each of the one or more objects within the second frame of the video is associated with a score, and wherein the score is based on characteristics of each of the one or more objects.
9. The method of claim 1 , further comprising:
calculating, by the video preprocessor of the mobile device, using the region data associated with the first region of the second frame of the video, frame boundaries to apply to the second frame of the video based on the aspect ratio of the mobile device and the second display orientation to display the second frame of the video;
identifying, by the video preprocessor of the mobile device, cropping coordinates for the cropped size centered on the first region of the second frame of the video based on the calculated frame boundaries; and
wherein cropping the second frame of the video is based on the cropping coordinates for the cropped size centered on the first region of the second frame of the video.
10. The method of claim 9 , wherein the cropped size is a rectangle centered on the first region of the second frame of the video.
11. A system comprising:
a mobile device comprising a memory, one or more processors, and a video preprocessor, the mobile device configured to:
receive data of a first frame of a video;
display, on a display of the mobile device, the first frame of the video in a first display orientation;
receive data of a second frame of the video;
detect that the mobile device has been rotated to display the video in the second display orientation instead of the first display orientation; and
responsive to detecting that the mobile device has been rotated, the video preprocessor of the mobile device is configured to:
extract, from the data of the second frame, region data associated with a first region of the second frame of the video;
crop the second frame of the video to a cropped size centered on the first region of the second frame and further based on an aspect ratio of the mobile device and a second display orientation to display the second frame of the video, the second display orientation is different from the first display orientation and is another of a portrait orientation or a landscape orientation; and
display, on the display of the mobile device, the cropped second frame of the video in the second display orientation.
12. The system of claim 11 , wherein the aspect ratio is a first aspect ratio, and wherein the video preprocessor of the mobile device is further configured to:
compare a second aspect ratio of the video to the first aspect ratio of the mobile device positioned in the second display orientation to display the second frame of the video, the second aspect ratio different from the first aspect ratio; and
determine, based on the first aspect ratio, to crop the second frame of the video to the cropped centered on the first region of the first frame and further based on the aspect ratio of the mobile device and the second display orientation to display the second frame of the video.
13. The system of claim 11 , wherein the region data is first region data, and wherein the mobile device is further configured to:
receive data of a third frame of the video; and
responsive to receiving the data of the third frame of the video, the video preprocessor of the mobile device is further configured to:
extract, from the data of the third frame, second region data associated with a second region of the third frame of the video, the second region is different from the first region of the second frame of the video;
crop the third frame of the video to a cropped size centered on the second region of the third frame and further based on the aspect ratio of the mobile device and the second display orientation to display the third frame of the video; and
display, on the display of the mobile device, the cropped third frame of the video in the second display orientation.
14. The system of claim 11 , wherein the region data is first region data, and wherein the mobile device is further configured to:
receive data of a third frame of the video;
detect that the mobile device has been rotated to display the video in the first display orientation instead of the second display orientation; and
display, on the display of the mobile device, responsive to receiving the data of the third frame of the video, the third frame of the video in the first display orientation.
15. The system of claim 14 , wherein the aspect ratio is a first aspect ratio, and wherein the video preprocessor of the mobile device is further configured to:
compare a second aspect ratio of the video to a third aspect ratio of the mobile device positioned in the first display orientation to display the third frame of the video; and
determine, based on the second aspect ratio being less than the third aspect ratio, to display, on the display of the mobile device, the third frame of the video in the first display orientation.
16. The system of claim 11 , wherein the first frame of the video is displayed on the display of the mobile device responsive to the mobile device receiving the data of the first frame of the video.
17. The system of claim 11 , wherein responsive to detecting that the mobile device has been rotated to display the video in the second display orientation instead of the first display orientation, the video preprocessor of the mobile device is further configured to:
identify, from the data of the second frame, one or more objects within the second frame to center on;
determine, based on the identified one or more objects within the second frame, the first region of the second frame of the video for extraction; and
wherein the extracted region data is associated with the first region determined based on the identified one or more objects.
18. The system of claim 17 , wherein each of the one or more objects within the second frame of the video is associated with a score, and wherein the score is based on characteristics of each of the one or more objects.
19. The system of claim 11 , wherein the video preprocessor of the mobile device is further configured to:
calculate, using the region data associated with the first region of the second frame of the video, frame boundaries to apply to the second frame of the video based on the aspect ratio of the mobile device and the second display orientation to display the second frame of the video;
identify cropping coordinates for the cropped size centered on the first region of the second frame of the video based on the calculated frame boundaries; and
wherein cropping the second frame of the video is based on the cropping coordinates for the cropped size centered on the first region of the second frame of the video.
20. The system of claim 19 , wherein the cropped size is a rectangle centered on the first region of the second frame of the video.