Motion estimation through the use of on-camera sensor information
Systems and methods for utilizing on-camera sensor information to improve video and/or image encoding quality are discussed herein. Specifically, the systems and methods may utilize on-camera sensor information to efficiently determine whether to encode a particular frame within a set of frames as an intra frame. When captured on video, a particular arrangement of a group of pixels within a frame may comprise a visual representation of an object within the frame. When encoding video footage, motion information characterizing motion of an image capturing device over time may be used to predict displacement of an arrangement of group of pixels between frames. These predictions may be used to efficiently determine whether to encode a particular frame as an intra frame.
1. A system that utilizes sensor information to improve motion estimation in video and/or image encoding, the system comprising:
one or more physical computer processors configured by computer readable instructions to:
obtain video information related to video content, the video content comprising a plurality of frames captured in a sequence by an image capturing device, wherein the plurality of frames comprises a sequence of frames comprising at least a first frame captured at a first time and a second frame captured at a second time;
determine an initial region of the first frame within which a visual representation of an object is located within the first frame, the initial region comprising less than a total area of the first frame;
obtain motion information characterizing motion of the image capturing device between the first time and the second time, wherein the motion information is generated by one or more motion sensors associated with the image capturing device;
identify a different region of the second frame as a search area in which to search for the visual representation of the object within the second frame based on the motion of the image capturing device between the first time and the second time and the initial region of the visual representation of the object within the first frame, wherein the different region comprises less than a total area of the second frame, and wherein identification of the different region includes determination of a change in spatial areas of the first frame and the second frame within which the visual representation of the object is located from the initial region of the first frame to the different region of the second frame based on the motion of the image capturing device between the first time and the second time;
search for the visual representation of the object within the different region of the second frame;
responsive to a determination that the visual representation of the object is located within the different region of the second frame, encode the second frame as an inter frame; and
responsive to a determination that the visual representation of the object is not located within the different region of the second frame, encode the second frame as an intra frame;
wherein encoding of the second frame as the inter frame or the intra frame based on the determination of whether or not the visual representation of the object is located within the different region of the second frame enables encoding of the second frame without analysis of entirety of the second frame.
2. The system of claim 1 , wherein the first frame and the second frame are adjacent frames within the plurality of frames.
3. The system of claim 1 , wherein the one or more motion sensors include one or more of an inertial measurement unit, a GPS component, an accelerometer, a gyroscope, an altimeter, and/or a distance measurement sensor.
4. The system of claim 1 , wherein the motion information characterizing motion of the image capturing device includes one or more of zoom, translational motion, rotational motion, vertical motion, horizontal motion, acceleration motion, and/or deceleration motion.
5. The system of claim 1 , wherein identifying the different region of the second frame as the search area includes determining a location of the different region within the second frame.
6. The system of claim 1 , wherein identifying the different region of the second frame as the search area includes identifying one or more dimensions of the different region within the second frame.
7. The system of claim 1 , wherein encoding the second frame as the inter frame comprises encoding the second frame by referencing one or more previous frames within the sequence of frames.
8. A method of utilizing sensor information to improve motion estimation in video and/or image encoding, the method comprising:
obtaining video information related to video content, the video content comprising a plurality of frames captured in a sequence by an image capturing device, wherein the plurality of frames comprises a sequence of frames comprising at least a first frame captured at a first time and a second frame captured at a second time;
determining an initial region of the first frame within which a visual representation of an object is located within the first frame, the initial region comprising less than a total area of the first frame;
obtaining motion information characterizing motion of the image capturing device between the first time and the second time, wherein the motion information is generated by one or more motion sensors associated with the image capturing device;
identifying a different region of the second frame as a search area in which to search for the visual representation of the object within the second frame based on the motion of the image capturing device between the first time and the second time and the initial region of the visual representation of the object within the first frame, wherein the different region comprises less than a total area of the second frame, and wherein identification of the different region includes determination of a change in spatial areas of the first frame and the second frame within which the visual representation of the object is located from the initial region of the first frame to the different region of the second frame based on the motion of the image capturing device between the first time and the second time;
searching for the visual representation of the object within the different region of the second frame;
responsive to a determination that the visual representation of the object is located within the different region of the second frame, encoding the second frame as an inter frame; and
responsive to a determination that the visual representation of the object is not located within the different region of the second frame, encoding the second frame as an intra frame;
wherein encoding of the second frame as the inter frame or the intra frame based on the determination of whether or not the visual representation of the object is located within the different region of the second frame enables encoding of the second frame without analysis of entirety of the second frame.
9. The method of claim 8 , wherein the first frame and the second frame are adjacent frames within the plurality of frames.
10. The method of claim 8 , wherein the one or more motion sensors include one or more of an inertial measurement unit, a GPS component, an accelerometer, a gyroscope, an altimeter, and/or a distance measurement sensor.
11. The method of claim 8 , wherein the motion information characterizing motion of the image capturing device includes one or more of zoom, translational motion, rotational motion, vertical motion, horizontal motion, acceleration motion, and/or deceleration motion.
12. The method of claim 8 , wherein identifying the different region of the second frame as the search area includes determining a location of the different region within the second frame.
13. The method of claim 8 , wherein identifying the different region of the second frame as the search area includes identifying one or more dimensions of the different region within the second frame.
14. The method of claim 8 , wherein encoding the second frame as the inter frame comprises encoding the second frame by referencing one or more previous frames within the sequence of frames.