Using motion information to assist in image processing
View Patent ↗Embodiments of the present application use motion information of an imaging device in encoding video or image frames captured by the imaging device. In one embodiment, a video encoding system that processes image data captured by an image sensor comprises at least one encoder that receives motion information describing motion of the imaging sensor during capturing of the image data and that encodes the image data with assistance of the motion information.
1. An encoding system comprising:
processing circuitry configured to perform encoding of image data, the encoding including a plurality of encoding sub-processes;
imaging circuitry configured to capture and deliver raw image data to the processing circuitry;
second circuitry, independent of the imaging circuitry, configured to deliver motion related information to the processing circuitry, the motion related information being generated at least near a time of capture by the imaging circuitry of the raw image data and being derived independently of analysis of the raw image data; and
the processing circuitry is configured to modify at least one of the plurality of encoding sub-processes of the raw image data using the motion related information that was derived independently of the analysis of the raw image data, wherein the at least one of the plurality of encoding sub-processes comprises compressing the raw image data.
2. The encoding system of claim 1 , the plurality of encoding sub-processes comprising:
a process to correct motion blur detected in the raw image data, wherein the process derives a point spread function of motion that caused the motion blur by projecting an infrared point onto a scene that is captured by secondary imaging circuitry.
3. The encoding system of claim 1 , wherein the at least one of the plurality of encoding sub-processes searches neighboring frames of the raw image data for a current block of image data and excludes at least one neighboring frame from being searched based on the motion related information.
4. The encoding system of claim 1 , wherein the at least one of the plurality of encoding sub-processes, based on the motion related information, constrains a search for a motion vector in a particular area of the raw image data captured during movement of an image sensor that captured the raw image data.
5. The encoding system of claim 1 , wherein the at least one of the plurality of encoding sub-processes classifies motion detected during capturing of the raw image data and uses the classification to accelerate encoding of the raw image data.
6. The encoding system of claim 5 , wherein the classification characterizes a nature of the motion, the classification comprising at least one of zooming, panning, or vibration.
7. The encoding system of claim 1 , the second circuitry comprising:
a motion sensory component configured to detect motion of an image sensor that captured the raw image data and configured to output the motion related information,
wherein the motion related information is separate from motion estimation data.
8. The encoding system of claim 7 , wherein the motion sensory component comprises one of an accelerometer or a gyroscope.
9. The encoding system of claim 7 , wherein the motion sensory component comprises image stabilization circuitry or zoom control circuitry for an optical lens configured to be used in capturing the raw image data.
10. An encoding device comprising:
processing circuitry configured to perform encoding of image data, the encoding including a plurality of encoding sub-processes;
interface circuitry through which the processing circuitry is configured to receive both raw image data and motion related information, receipt of the raw image data and the motion related information having at least some time synchrony, wherein the motion related information is derived independently of analysis of the raw image data;
the processing circuitry is configured to perform the encoding of the raw image data; and
the processing circuitry, using the motion related information, is configured to modify at least one of the plurality of encoding sub-processes of the raw image data,
wherein at least one of the plurality of encoding sub-processes uses the motion related information that was derived independently of the analysis of the raw image data to facilitate a motion estimation process that is part of the encoding of the raw image data.
11. The encoding device of claim 10 , the plurality of encoding sub-processes comprising:
a process to correct motion blur detected in the raw image data, wherein the process derives a point spread function of motion that caused the motion blur by projecting an infrared point onto a scene that is captured by secondary imaging circuitry.
12. The encoding device of claim 10 , wherein the at least one of the plurality of encoding sub-processes searches neighboring frames of the raw image data for a current block of image data and excludes at least one neighboring frame from being searched based on the motion related information.
13. The encoding device of claim 10 , further comprising an image sensor configured to capture the raw image data.
14. The encoding device of claim 10 , the interface circuitry comprising:
a motion sensory component configured to detect motion of an image sensor that captured the raw image data and configured to output the motion related information.
15. The encoding device of claim 14 , wherein the motion sensory component comprises one of an accelerometer or a gyroscope.
16. The encoding device of claim 14 , wherein the motion sensory component comprises one of image stabilization circuitry or zoom control circuitry for an optical lens configured to be used in capturing the raw image data.
17. The encoding device of claim 10 , wherein the at least one of the plurality of encoding sub-processes classifies motion detected during capturing of the raw image data and uses the classification to accelerate encoding of the raw image data.
18. The encoding device of claim 17 , wherein the at least one of the plurality of encoding sub-processes modifies an encoding frame rate using the classification and responsive to the motion being classified.
19. The encoding device of claim 17 , wherein the at least one of the plurality of encoding sub-processes modifies an encoding resolution using the classification and responsive to the motion being classified.
20. A method comprising
encoding of image data by processing circuitry, the encoding including a plurality of encoding sub-processes;
capturing and delivering raw image data to the processing circuitry;
delivering motion related information to the processing circuitry, the motion related information being generated at least near a time of capture of the raw image data and being derived independently of analysis of the raw image data; and
using the motion related information to modify at least one of the plurality of encoding sub-processes of the raw image data,
wherein at least one of the plurality of encoding sub-processes comprises classifying a type of motion detected during capturing of the raw image data and using the classification to facilitate acceleration of compressing of the raw image data.