Methods and systems for accelerated image rendering with motion compensation
Various methods and systems are provided for accelerated image rendering with motion compensation. In one embodiment, a method comprises calculating motion between a preceding image frame and a target image frame to be rendered, rendering a small image with a size smaller than a target size of the target image frame based on the calculated motion, and generating the target image frame at the target size based on the small image, the calculated motion, and a reference image frame. In this way, high-quality image frames for a video stream may be generated with a reduced amount of rendering for each frame, thereby reducing the overall processing resources dedicated to rendering as well as the power consumption for image rendering.
1. A method, comprising:
determining a motion between a preceding image frame of a sequence of image frames to be rendered and a reference image frame;
rendering a small image for the preceding image frame, the rendered image having a resolution lower than the reference image frame;
upscaling the rendered image to a target resolution of the reference image frame;
detecting one or more areas of the rendered image to be blended;
adjusting brightness of the pixels of a motion-compensated stored reference image frame based on a brightness of the pixels of the rendered image; and
generating an output image frame at a target size based on both the rendered image and the motion-compensated stored reference image frame corresponding with a preceding output image frame,
wherein, within the one or more areas of the rendered image to be blended, pixels of the output image frame are the corresponding pixels of the rendered image blended with the corresponding pixels of the motion-compensated stored reference image frame; and
wherein, outside of the one or more areas of the rendered image to be blended, pixels of the output image frame are the corresponding pixels of the rendered image or the reference image frame.
2. The method of claim 1 , wherein generating the output image frame at the target size comprises performing motion compensation of a stored reference image frame based on motion between the stored reference image frame and the rendered image.
3. The method of claim 2 , wherein the stored reference image frame corresponds with the most recent preceding output image frame.
4. The method of claim 1 , wherein the detecting of the one or more areas of the rendered image to be blended includes detecting pixel-level differences between pixels of a rendered image and pixels of the motion-compensated stored reference image frame.
5. The method of claim 1 , wherein the motion between the preceding image frame and reference image frame is below a threshold amount of motion.
6. A method, comprising:
determining a motion between a preceding image frame of a sequence of image frames to be rendered and a reference image frame;
rendering a small image for the preceding image frame, the rendered image having a resolution lower than the reference image frame;
upscaling the rendered image to a target resolution of the reference image frame;
detecting one or more areas of the rendered image to be blended;
adjusting brightness of the pixels of a motion-compensated stored reference image frame based on a brightness of the pixels of the rendered image;
generating an output image frame at a target size based on the rendered image and the motion-compensated stored reference image frame corresponding with a preceding output image frame; and
filling regions of the output image frame away from the rendered image with pixels of the motion-compensated stored reference image frame responsive to the determined motion not indicating a scene change or a complex motion,
wherein, within the one or more areas of the rendered image to be blended, pixels of the output image frame are the corresponding pixels of the rendered image blended with the corresponding pixels of the motion-compensated stored reference image frame; and
wherein, outside of the one or more areas of the rendered image to be blended, pixels of the output image frame are the corresponding pixels of the rendered image or the reference image frame.
7. The method of claim 6 , further comprising generating the motion-compensated stored reference image frame by applying motion compensation to a stored reference image frame based on motion between the rendered image and the stored reference image frame.
8. The method of claim 6 , further comprising storing the output image frame in a memory for use as a reference image frame for at least one subsequent output image frame.
9. The method of claim 6 , wherein the detecting of the one or more areas of the rendered image to be blended includes detecting pixel-level differences between pixels of the rendered image and pixels of the motion-compensated stored reference image frame.
10. A system, comprising:
an image processor configured with instructions in non-transitory memory that when executed cause the image processor to:
determine a motion between a preceding image frame of a sequence of image frames to be rendered and a reference image frame;
rendering a small image for the preceding image frame, the rendered image having a resolution lower than the reference image frame;
upscaling the rendered image to a target resolution of the reference image frame;
detect one or more areas of the rendered image to be blended;
adjusting brightness of the pixels of a motion-compensated stored reference image frame based on a brightness of the pixels of the rendered image; and
generate an output image frame at a target size based on both the rendered image and the motion-compensated stored reference image frame corresponding with a preceding output image frame,
wherein, within the one or more areas of the rendered image to be blended, pixels of the output image frame are the corresponding pixels of the rendered image blended with the corresponding pixels of the motion-compensated stored reference image frame; and
wherein, outside of the one or more areas of the rendered image to be blended, pixels of the output image frame are the corresponding pixels of the rendered image or the reference image frame.
11. The system of claim 10 , wherein generating the output image frame at the target size comprises performing motion compensation of a stored reference image frame based on motion between the stored reference image frame and the rendered image.
12. The system of claim 11 , wherein the stored reference image frame corresponds with the most recent preceding output image frame.
13. The system of claim 10 , wherein the detecting of the one or more areas of the rendered image to be blended is based on a motion vector field.
14. The system of claim 10 , wherein the detecting of the one or more areas of the rendered image to be blended includes detecting pixel-level differences between pixels of the rendered image and pixels of the motion-compensated stored reference image frame.