Motion compensation for neural network enhanced images

1. A device comprising:

a memory configured to store an enhanced image; and

one or more processors configured to:

apply a neural network to a first image to generate the enhanced image;

adjust at least a portion of a high-frequency component of the enhanced image, based on a motion associated with the first image being greater than a threshold motion, to generate an adjusted high-frequency image component; and

combine a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

2. The device of claim 1 , wherein the one or more processors are configured to use a temporal filter based on a previous image to adjust at least the portion of the high-frequency component of the enhanced image.

3. The device of claim 2 , wherein when the one or more processors are configured to process the first image, the motion compensation operation and the temporal filter operate based on a most recently generated previous adjusted enhanced image.

4. The device of claim 1 , wherein the one or more processors are configured to:

generate a low-resolution image of an input image;

encode the low-resolution image to generate image encoding data; and

local decode the image encoding data to generate the first image, the first image corresponding to a decoded low-resolution image.

5. The device of claim 4 , wherein the one or more processors are configured to:

generate refinement data based on a comparison between the adjusted enhanced image and the input image; and

initiate transmission of the image encoding data and the refinement data to a second device.

6. The device of claim 5 , wherein the one or more processors are configured to:

generate a refinement image based on the comparison between the adjusted enhanced image and the input image;

apply a transform to the refinement image to generate a transformed refinement image;

apply quantization to the transformed refinement image to generate quantized refinement data; and

apply entropy coding to the quantized refinement data to generate the refinement data.

7. The device of claim 1 , wherein the one or more processors are configured to decode image encoding data to generate the first image, the first image corresponding to a decoded low-resolution image.

8. The device of claim 7 , further comprising a modem configured to receive a bitstream that includes the image encoding data.

9. The device of claim 1 , wherein the one or more processors are configured to apply refinement data to the adjusted enhanced image to generate a refined enhanced image.

10. The device of claim 9 , further comprising a modem configured to receive a bitstream that includes the refinement data.

11. The device of claim 1 , further comprising a display device configured to display an output image that is based on the adjusted enhanced image.

12. The device of claim 11 , wherein the display device includes a mixed reality (MR) display device, a virtual reality (VR) display device, an augmented reality (AR) display device, or a combination thereof.

13. The device of claim 1 , wherein the one or more processors are configured to execute the instructions to receive the first image from a camera.

14. The device of claim 1 , wherein the one or more processors are configured to receive the first image from a graphic processing unit (GPU).

15. The device of claim 1 , wherein the one or more processors are configured to execute the instructions to retrieve the first image from one of a camera buffer or a graphics processing unit (GPU) buffer.

16. The device of claim 1 , wherein the one or more processors are configured to:

receive an image enhancement request from an application; and

retrieve, based on a type of the application, the first image from one of a camera buffer or a graphics processing unit (GPU) buffer.

17. A method comprising:

applying, at a device, a neural network to a first image to generate an enhanced image;

adjusting, at a device, at least a portion of a high-frequency component of the enhanced image, based on a motion associated with the first image being greater than a threshold motion, to generate an adjusted high-frequency image component; and

combining, at the device, a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

18. The method of claim 17 , further comprising using, at the device, a temporal filter based on a previous image to adjust at least the portion of the high-frequency component of the enhanced image.

19. The method of claim 18 , wherein during processing of the first image, the motion compensation operation and the temporal filter operate based on a most recently generated previous adjusted enhanced image.

20. The method of claim 17 , further comprising:

generating, at the device, a low-resolution image of an input image;

encoding, at the device, the low-resolution image to generate image encoding data; and

local decoding, at the device, the image encoding data to generate the first image, the first image corresponding to a decoded low-resolution image.

21. The method of claim 20 , further comprising:

generating, at the device, refinement data based on a comparison between the adjusted enhanced image and the input image; and

initiating transmission of the image encoding data and the refinement data from the device to a second device.

22. The method of claim 17 , further comprising decoding, at the device, image encoding data to generate the first image, the first image corresponding to a decoded low-resolution image.

23. The method of claim 17 , further comprising applying, at the device, refinement data to the adjusted enhanced image to generate a refined enhanced image.

24. The method of claim 17 , further comprising retrieving, at the device, the first image from one of a camera buffer or a graphics processing unit (GPU) buffer.

25. The method of claim 17 , further comprising:

receiving, at the device, an image enhancement request from an application; and

retrieving, based on a type of the application, the first image from one of a camera buffer or a graphics processing unit (GPU) buffer.

26. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:

apply a neural network to a first image to generate an enhanced image;

adjust at least a portion of a high-frequency component of the enhanced image, based on a motion associated with the first image being greater than a threshold motion, to generate an adjusted high-frequency image component; and

combine a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

27. The non-transitory computer-readable medium of claim 26 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to use a temporal filter based on a previous image to adjust at least the portion of the high-frequency component of the enhanced image.

28. An apparatus comprising:

means for applying a neural network to a first image to generate an enhanced image;

means for adjusting at least a portion of a high-frequency component of the enhanced image, based on a motion associated with the first image being greater than a threshold motion, to generate an adjusted high-frequency image component; and

means for combining a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

29. The apparatus of claim 28 , wherein the means for applying, the means for adjusting, and the means for combining are integrated into at least one of a portable electronic device, a car, a vehicle, a camera, a computing device, a communication device, an internet-of-things (IoT) device, a virtual reality (VR) device, a mixed reality (MR) device, an augmented reality (AR) device, or a combination thereof.