Adjusting the image of an object to search for during video encoding due to changes in appearance caused by camera movement
Systems and methods for utilizing on-device sensor information to improve video encoding quality are discussed herein. Specifically, the systems and methods may utilize on-device sensor information to efficiently determine whether to encode a particular frame within a set of frames as an intra frame. When captured on video, a particular arrangement of a group of pixels within a frame may comprise a visual representation of an object within the frame. When encoding video footage, motion information characterizing motion of an image capturing device over time may be used to predict the modification of an arrangement of a group of pixels between frames. These predictions may be used to efficiently determine whether to encode a particular frame as an intra frame.
1. A system for utilizing sensor information to improve motion estimation in video and/or image encoding, the system comprising:
one or more physical computer processors configured by computer readable instructions to:
identify a visual representation of an object within a first frame captured at a first time by an image capturing device, wherein the visual representation is represented within a first arrangement of pixels defining a first geometric shape of the object within the first frame;
obtain motion information characterizing motion of the image capturing device between the first time and a second time at which a second frame was captured by the image capturing device, wherein the motion information is generated by one or more motion sensors associated with the image capturing device;
determine a modified visual representation of the object to search for within the second frame based on the motion of the image capturing device between the first time and the second time at which the second frame was captured by the image capturing device and the visual representation of the object identified within the first frame, wherein the modified visual representation includes a different representation of the object from visual representation due to the motion of the image capturing device with respect to the object between the first time and the second time, the modified visual representation represented within a second arrangement of pixels different from the first arrangement of pixels, the second arrangement of pixels defining a second geometric shape of the object within the second frame, the second geometric shape different from the first geometric shape;
search for the modified visual representation within the second frame based on a search of the second geometric shape defined by the second arrangement of pixels within the second frame;
determine whether the modified visual representation is or is not located within the second frame based on the second geometric shape defined lithe second arrangement of pixels being found or not being found within the second frame;
responsive to a determination that the modified visual representation is located within the second frame, encode the second frame as an inter frame; and
responsive to a determination that the modified visual representation is not located within the second frame, encode the second frame as an intra frame.
2. The system of claim 1 , wherein the first frame and the second frame are adjacent frames within a sequence of frames.
3. The system of claim 1 , wherein the second time occurs before or after the first time.
4. The system of claim 1 , wherein the one or more motion sensors include one or more of an inertial measurement unit, a GPS component, an accelerometer, a gyroscope, an altimeter, and/or a distance measurement sensor.
5. The system of claim 1 , wherein the motion information characterizing motion of the image capturing device includes one or more of translational motion, rotational motion, vertical motion, horizontal motion, acceleration motion, and/or deceleration motion.
6. The system of claim 1 , wherein the first arrangement of pixels further defines a first perspective of the object within the first frame and the second arrangement of pixels further defines a second perspective of the object within the second frame, the second perspective different from the first perspective.
7. The system of claim 1 , wherein the first arrangement of pixels further defines a first size of the object within the first frame and the second arrangement of pixels further defines a second size of the object within the second frame, the second size different from the first size.
8. The system of claim 1 , wherein determining the modified visual representation of the object includes determining boundaries of the modified visual representation of the object within the second frame, the boundaries of the modified visual representation of the object including a portion of the second frame.
9. The system of claim 1 , wherein only a portion of the second frame is searched, the portion comprising less than a total area of the second frame.
10. The system of claim 9 , wherein responsive to the determination that the modified visual representation is not located within the portion of the second frame, the one or more physical computer processors are further configured to:
search for the modified visual representation within at least a second portion of the second frame; and
determine that the modified visual representation is not located within at least the second portion of the second frame, wherein the second frame is encoded as the intra frame based further on the determination that the modified visual representation is not located within at least the second portion of the second frame.
11. The system of claim 1 , wherein the one or more physical computer processors are further configured to:
generate compressed video information that includes at least the first frame, the second frame, and the motion information; and
decode the compressed video information based on the motion information.
12. A method for utilizing sensor information to improve motion estimation in video and/or image encoding, the method comprising:
identifying a visual representation of an object within a first frame captured at a first time by an image capturing device, wherein the visual representation is represented within a first arrangement of pixels defining a first geometric shape of the object within the first frame;
obtaining motion information characterizing motion of the image capturing device between the first time and a second time at which a second frame was captured by the image capturing device, wherein the motion information is generated by one or more motion sensors associated with the image capturing device;
determining a modified visual representation of the object to search for within the second frame based on the motion of the image capturing device between the first time and the second time at which the second frame was captured by the image capturing device and the visual representation of the object identified within the first frame, wherein the modified visual representation includes a different representation of the object from visual representation due to the motion of the image capturing device with respect to the object between the first time and the second time, the modified visual representation represented within a second arrangement of pixels different from the first arrangement of pixels, the second arrangement of pixels defining a second geometric shape of the object within the second frame, the second geometric shape different from the first geometric shape;
searching for the modified visual representation within the second frame based on a search of the second geometric shape defined by the second arrangement of pixels within the second frame;
determining whether the modified visual representation is or is not located within the second frame based on the second geometric shape defined by the second arrangement of pixels being found or not being found within the second frame; and
responsive to a determination that the modified visual representation is located within the second frame, encoding the second frame as an inter frame; and
responsive to a determination that the modified visual representation is not located within the second frame, encoding the second frame as an intra frame.
13. The method of claim 12 , wherein the first frame and the second frame are adjacent frames within a sequence of frames.
14. The method of claim 12 , wherein the second time occurs before or after the first time.
15. The method of claim 12 , wherein the one or more motion sensors include one or more of an inertial measurement unit, a GPS component, an accelerometer, a gyroscope, an altimeter, and/or a distance measurement sensor.
16. The method of claim 12 , wherein the motion information characterizing motion of the image capturing device includes one or more of translational motion, rotational motion, vertical motion, horizontal motion, acceleration motion, and/or deceleration motion.
17. The method of claim 12 , wherein the first arrangement of pixels further defines a first perspective of the object within the first frame and the second arrangement of pixels further defines a second perspective of the object within the second frame, the second perspective different from the first perspective.
18. The method of claim 12 , wherein the first arrangement of pixels further defines a first size of the object within the first frame and the second arrangement of pixels further defines a second size of the object within the second frame, the second size different from the first size.
19. The method of claim 12 , wherein determining the modified visual representation of the object includes determining boundaries of the modified visual representation of the object within the second frame, the boundaries of the modified visual representation of the object including a portion of the second frame.
20. The method of claim 12 , wherein only a portion of the second frame is searched, the portion comprising less than a total area of the second frame.
21. The method of claim 20 , wherein responsive to the determination that the modified visual representation is not located within the portion of the second frame, further comprising:
searching for the modified visual representation within at least a second portion of the second frame; and
determining that the modified visual representation is not located within at least the second portion of the second frame, wherein the second frame is encoded as the intra frame based further on the determination that the modified visual representation is not located within at least the second portion of the second frame.
22. The method of claim 12 , the method further comprising:
generating compressed video information that includes at least the first frame, the second frame, and the motion information; and
decoding the compressed video information based on the motion information.