Automatically cropping of landscape videos
Example systems, computer readable medium and methods for automatically generating portrait videos from landscape videos include receiving a landscaped video, identifying one or more objects to track, and tracking the one or more objects by moving a cropping window. A user interface is presented that provides options for a user to automatically convert a landscape video into a portrait video. The options include selecting one or more objects in the landscape video to track and tracking based on a flight plan used to capture the video. The automatic tracking uses a hierarchy of a person for tracking where a face is used if the person is close, an upper body is used if the person is farther away, and a whole body is used if the person is even farther away. A hierarchy of a person includes indications of which parts of the hierarchy to exclude first.
1. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, configure the at least one processor to perform operations comprising:
receiving a selection to generate a portrait video from a landscape video;
identifying a person in the landscape video for inclusion in the portrait video;
selecting, based on the person, a cropping window of a first portion of the landscape video to include in a first portion of the portrait video;
determining whether a face of the person can be identified;
in response to not being able to identify the face of the person,
identifying an upper body of the person associated with the person; and
in response to movement of the upper body of the person in the first portion of the landscape video, moving the cropping window left or moving the cropping window right to maintain the upper body of the person in the cropping window.
2. The system of claim 1 wherein the operations further comprise:
including the moved cropping window of a second portion of the landscape video in a corresponding second portion of the portrait video.
3. The system of claim 1 wherein the operations further comprise:
in response to being unable to identify the upper body of the person,
identifying a whole body of the person associated with the upper body of the person;
in response to movement of the whole body of the person in the third first portion of the landscape video, moving the cropping window left or moving the cropping window right to maintain the whole body in the cropping window; and
including the moved cropping window of a second portion of the landscape video in a corresponding second portion of the portrait video.
4. The system of claim 1 wherein the identifying further comprises:
identifying a plurality of people in the landscape video; and
selecting an upper body of a center person of the plurality of people as the person.
5. The system of claim 1 wherein the identifying further comprises:
identifying a body of a person in a center portion of the landscape video; and
selecting the body of the person as the upper body of the person.
6. The system of claim 1 wherein the identifying further comprises:
identifying a body of the person, and wherein the selecting further comprises:
selecting, based on centering the body of the person, the cropping window.
7. The system of claim 1 wherein the identifying further comprises:
identifying a center of a group of faces, the group of faces comprising a face of the person, and wherein the selecting further comprises:
selecting, based on centering the group of faces, the cropping window of the first portion of the landscape video to include in the first portion of the portrait video.
8. The system of claim 1 wherein the first portion is one or more frames of the landscape video.
9. The system of claim 1 wherein the identifying is performed with a neural network trained to identify people or wherein the identifying is based on 2-dimensional body tracking and segmentation masks.
10. The system of claim 1 wherein the identifying further comprises:
identifying a body of an animal or object, and wherein the selecting further comprises:
selecting, based on centering the body of the animal or object and the person, the cropping window of the first portion of the landscape video to include in the first portion of the portrait video.
11. The system of claim 1 wherein the portrait video has an aspect ratio of 9 to 16 and the landscape video has an aspect ratio of 4 to 3.
12. The system of claim 1 wherein the identifying further comprises:
identifying the person in the landscape video based on a selection received from a user of an area for identifying the person within the landscape video.
13. The system of claim 1 wherein the identifying further comprises:
identifying the person in the landscape video based on a flight plan associated with capturing the landscape video, wherein the flight plan indicates a location of the person.
14. The system of claim 1 wherein the operations further comprise:
after the selecting, processing the first portion of the portrait video to include the cropping window of the first portion of the landscape video.
15. The system of claim 1 , wherein the operations further comprise:
in response to being able to identify the face of the person, and to movement of the face in the first portion of the landscape video, moving the cropping window left or moving the cropping window right to maintain the face of the person in the cropping window.
16. A method comprising:
receiving, by at least one processor, a selection to generate a portrait video from a landscape video;
identifying a person in the landscape video for inclusion in the portrait video;
selecting, based on the person, a cropping window of a first portion of the landscape video to include in a first portion of the portrait video;
determining whether a face of the person can be identified;
in response to not being able to identify the face of the person,
identifying an upper body of the person associated with the person; and
in response to movement of the upper body of the person in the first portion of the landscape video, moving the cropping window left or moving the cropping window right to maintain the upper body of the person in the cropping window.
17. The method of claim 16 further comprising:
including the moved cropping window of a second portion of the landscape video in a corresponding second portion of the portrait video.
18. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of an apparatus of a system, cause the at least one processor to perform operations comprising:
receiving a selection to generate a portrait video from a landscape video;
identifying a person in the landscape video for inclusion in the portrait video;
selecting, based on the person, a cropping window of a first portion of the landscape video to include in a first portion of the portrait video;
determining whether a face of the person can be identified;
in response to not being able to identify the face of the person,
identifying an upper body of the person associated with the person; and
in response to movement of the upper body of the person in the first portion of the landscape video, moving the cropping window left or moving the cropping window right to maintain the upper body of the person in the cropping window.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the operations further comprise:
including the moved cropping window of a second portion of the landscape video in a corresponding second portion of the portrait video.
20. The non-transitory computer-readable storage medium of claim 18 , wherein the identifying is performed with a neural network trained to identify people or wherein the identifying is based on 2-dimensional body tracking and segmentation masks.