Motion deblurring using neural network architectures
This disclosure relates to improved techniques for performing computer vision functions including motion deblurring functions. The techniques described herein utilize a neural network architecture to perform these functions. The neural network architecture can include a human-aware attention model that is able to distinguish between foreground human objects and background portions of degraded images affected by motion blur. The neural network architecture further includes an encoder-decoder network that separately performs motion deblurring functions on foreground and background portions of degraded images, and reconstructs enhanced images corresponding to the degraded images.
1. A system for generating an enhanced image from a degraded image comprising:
one or more computing devices comprising one or more processors and one or more non-transitory storage devices for storing instructions, wherein execution of the instructions by the one or more processors causes the one or more computing devices to:
receive, at a neural network architecture, a degraded image that includes motion blur, the degraded image comprising a foreground portion and a background portion;
analyze, using a human-aware attention model associated with the neural network architecture, the degraded image to generate one or more human-aware attention masks;
generate, using an encoder associated with the neural network architecture, an encoded representation of the degraded image;
generate, using the encoded representation and the one or more human-aware attention masks, foreground feature information and background feature information corresponding to the degraded image;
receive, at a multi-branch decoder of the neural network architecture, the foreground feature information and the background feature information;
separately perform, with the multi-branch decoder of the neural network architecture, deblurring functions on the foreground portion and the background portion of the degraded image using the foreground feature information and the background feature information; and
reconstruct an enhanced image corresponding to the degraded image.
2. The system of claim 1 , wherein the multi-branch decoder comprises:
a foreground decoding branch that is configured to generate foreground deblurring information corresponding to the foreground portion of the degraded image;
a background decoding branch that is configured to generate background deblurring information corresponding to the background portion of the degraded image; and
a global decoding branch that is configured to generate global deblurring information corresponding to the degraded image.
3. The system of claim 2 , wherein the foreground deblurring information, the background deblurring information, and the global deblurring information is fused or combined to reconstruct the enhanced image.
4. The system of claim 3 , wherein combining the foreground deblurring information, the background deblurring information and the global deblurring information to reconstruct the enhanced image includes integrating at least a portion of the foreground deblurring information and at least a portion of the background deblurring information into the global deblurring information of the global decoding branch.
5. The system of claim 1 , wherein at least a portion of the neural network architecture is trained using a training data set that comprises:
training images that include annotations identifying human objects in foreground portions of training images.
6. The system of claim 5 , wherein at least a portion of the training images are synthesized, at least in part, by averaging sequential frames from a video to simulate a degradation process.
7. The system of claim 1 , wherein the encoded representation comprises global feature information corresponding to both the foreground portion and background portion of the degraded image.
8. The system of claim 1 , wherein the system is integrated directly into an image capturing device.
9. The system of claim 1 , wherein the system is integrated, or communicates, with one or more of: a surveillance system, a facial recognition system, an automated vehicular system, or a social media application.
10. The system of claim 1 , wherein:
the motion blur associated with the degraded image includes distortions resulting from both object movement and movement of an image capturing device; and
the neural network architecture is trained to remove the distortions resulting from both object movement and movement of an image capturing device.
11. A method for generating an enhanced image from a degraded image comprising:
receiving, at a neural network architecture, a degraded image that includes motion blur, the degraded image comprising a foreground portion and a background portion;
analyzing, using a human-aware attention model associated with the neural network architecture, the degraded image to generate one or more human-aware attention masks;
generating, using an encoder associated with the neural network architecture, an encoded representation of the degraded image;
generating, using the encoded representation and the one or more human-aware attention masks, foreground feature information and background feature information corresponding to the degraded image;
receiving, at a multi-branch decoder of the neural network architecture, the foreground feature information and the background feature information;
separately performing, with the multi-branch decoder of the neural network architecture, deblurring functions on the foreground portion and the background portion of the degraded image using the foreground feature information and the background feature information; and
reconstructing an enhanced image corresponding to the degraded image.
12. The method of claim 11 , wherein the multi-branch decoder comprises:
a foreground decoding branch that is configured to generate foreground deblurring information corresponding to the foreground portion of the degraded image;
a background decoding branch that is configured to generate background deblurring information corresponding to the background portion of the degraded image; and
a global decoding branch that is configured to generate global deblurring information corresponding to the degraded image.
13. The method of claim 12 , wherein the foreground deblurring information, the background deblurring information and the global deblurring information is fused or combined to reconstruct the enhanced image.
14. The method of claim 13 , wherein combining the foreground deblurring information, the background deblurring information and the global deblurring information to reconstruct the enhanced image includes integrating at least a portion of the foreground deblurring information and at least a portion of the background deblurring information into the global deblurring information of the global decoding branch.
15. The method of claim 11 , wherein at least a portion of the neural network architecture is trained using a training data set that comprises:
training images that include annotations identifying human objects in foreground portions of training images.
16. The method of claim 15 , wherein at least a portion of the training images are synthesized, at least in part, by averaging sequential frames from a video to simulate a degradation process.
17. The method of claim 11 , wherein the encoded representation comprises global feature information corresponding to both the foreground portion and background portion of the degraded image.
18. The method of claim 11 , wherein the system is integrated directly into an image capturing device.
19. The method of claim 11 , wherein:
the motion blur associated with the degraded image includes distortions resulting from both object movement and movement of an image capturing device; and
the neural network architecture is trained to remove the distortions resulting from both object movement and movement of an image capturing device.
20. A computer program product comprising a non-transitory computer-readable medium including instructions for causing a computer to:
receive, at a neural network architecture, a degraded image that includes motion blur, the degraded image comprising a foreground portion and a background portion;
analyze, using a human-aware attention model associated with the neural network architecture, the degraded image to generate one or more human-aware attention masks;
generate, using an encoder associated with the neural network architecture, an encoded representation of the degraded image;
generate, using the encoded representation and the one or more human-aware attention masks, foreground feature information and background feature information corresponding to the degraded image;
receive, at a multi-branch decoder of the neural network architecture, the foreground feature information and the background feature information;
separately perform, with the multi-branch decoder of the neural network architecture, deblurring functions on the foreground portion and the background portion of the degraded image using the foreground feature information and the background feature information; and
reconstruct an enhanced image corresponding to the degraded image.