Some embodiments remove an obstruction, e.g., a smudge, from a video. The video may be broken down into layers, and every pixel in space/time may be assigned a layer. Typically, the obstruction will be on one layer, and the background and foreground will be on other layers. Some embodiments detect which layer is generated by the smudge. Some embodiments use a motion model as a judging criteria for deciding the smudge layer.
1. A computer implemented image processing method of removing an obstruction from a video image stream, the method comprising:
obtaining an electronic representation of a video image stream comprising a plurality of pixels, each pixel of the plurality of pixels comprising a time and a location;
modeling, by at least one electronic processor, each of a plurality of video image layers as dynamic textures;
assigning, by at least one electronic parallel processor, each of the plurality of pixels to one of the plurality of video image layers, wherein each layer represents scene elements that are a certain distance from a camera image plane of the video image stream;
determining, by at least one electronic processor, a particular layer of the plurality of video image layers in which the obstruction occurs;
inpainting, by at least one electronic processor, on the particular layer, a region of the video image stream corresponding to the obstruction; and
providing the video stream subsequent to the video inpainting.
2. The method of claim 1 , wherein the obstruction comprises a smudge on a lens of an imaging system that produced the video image stream.
3. The method of claim 2 , wherein the lens is mounted outside a building, or the lens is mounted outside a vehicle.
4. The method of claim 3 , wherein the lens is mounted outside a submarine, and wherein a periscope of the submarine comprises the lens.
5. The method of claim 1 , wherein the modeling each of a plurality of video image layers as a dynamic texture comprises modeling each of a plurality of video image layers using a respective linear dynamic system.
6. The method of claim 1 , wherein the assigning each of the plurality of pixels to one of the plurality of video image layers comprises applying a layered dynamic texture model.
7. The method of claim 6 , wherein the assigning each of the plurality of pixels to one of the plurality of video image layers further comprises implementing a variational approximation of an expectation-maximization algorithm.
8. The method of claim 6 , wherein the applying the layered dynamic texture model comprises forming a Markov random field representing a plurality of discrete hidden variables.
9. The method of claim 1 ,
wherein the video image stream comprises a representation of a reticle, and
wherein the determining a particular layer of the plurality of video image layers in which the obstruction occurs comprises comparing a dynamics of a video image layer comprising the reticle to a dynamics of each of the plurality of video image layers.
10. The method of claim 1 , wherein the determining a particular layer of the plurality of video image layers in which the obstruction occurs comprises comparing each of the plurality of video image layers to a supervised learning regression model of motion.
11. The method of claim 1 , wherein the assigning, by at least one electronic parallel processor, each of the plurality of pixels to one of the plurality of video image layers comprises assigning, by at least one electronic video processor, each of the plurality of pixels to one of the plurality of video image layers.
12. The method of claim 1 , wherein the inpainting, on the particular layer, a region of the video image stream corresponding to the obstruction comprises, for each of a second plurality of pixels comprising the obstruction, identifying a corresponding candidate frame from the video image stream in which the pixel of the second plurality of pixels is not present.
13. An image processing system for removing an obstruction from a video image stream, the system comprising:
at least one electronic interface configured to obtain an electronic representation of a video image stream comprising a plurality of pixels, each pixel of the plurality of pixels comprising a time and a location;
at least one electronic processor configured to model each of a plurality of video image layers as dynamic textures;
at least one electronic parallel processor configured to assign each of the plurality of pixels to one of the plurality of video image layers, wherein each layer represents scene elements that are a certain distance from a camera image plane of the video image stream;
at least one electronic processor configured to determine a particular layer of the plurality of video image layers in which the obstruction occurs;
at least one electronic processor configured to inpaint on the particular layer, a region of the video image stream corresponding to the obstruction; and
at least one electronic interface configured to provide the video stream subsequent to the video inpainting.
14. The system of claim 13 , wherein the obstruction comprises a smudge on a lens of an imaging system that produced the video image stream.
15. The system of claim 14 , wherein the lens is mounted outside a building, or the lens is mounted outside a vehicle.
16. The system of claim 15 , wherein the lens is mounted outside a submarine, and wherein a periscope of the submarine comprises the lens.
17. The system of claim 13 , wherein the at least one electronic processor configured to model each of a plurality of video image layers as a dynamic texture comprises at least one electronic processor configured to model each of a plurality of video image layers using a respective linear dynamic system.
18. The system of claim 13 , wherein the at least one electronic parallel processor configured to assign each of the plurality of pixels to one of the plurality of video image layers comprises at least one electronic parallel processor configured to apply a layered dynamic texture model.
19. The system of claim 18 , wherein the at least one electronic parallel processor configured to assign each of the plurality of pixels to one of the plurality of video image layers further comprises at least one electronic parallel processor configured to implement a variational approximation of an expectation-maximization algorithm.
20. The system of claim 18 , wherein the at least one electronic parallel processor configured to implement the layered dynamic texture model comprises at least one electronic parallel processor configured to form a Markov random field representing a plurality of discrete hidden variables.
21. The system of claim 13 ,
wherein the video image stream comprises a representation of a reticle, and
wherein the at least one electronic processor configured to determine a particular layer of the plurality of video image layers in which the obstruction occurs comprises at least one electronic processor configured to compare a dynamics of a video image layer comprising the reticle to a dynamics of each of the plurality of video image layers.
22. The system of claim 13 , wherein the at least one electronic processor configured to determine a particular layer of the plurality of video image layers in which the obstruction occurs comprises at least one electronic processor configured to compare each of the plurality of video image layers to a supervised learning regression model of motion.
23. The system of claim 13 , wherein the at least one electronic parallel processor configured to assign each of the plurality of pixels to one of the plurality of video image layers comprises at least one electronic video processor configured to assign each of the plurality of pixels to one of the plurality of video image layers.
24. The system of claim 13 , wherein the at least one electronic processor configured to inpaint, on the particular layer, a region of the video image stream corresponding to the obstruction comprises, at least one electronic processor configured to identify, for each of a second plurality of pixels comprising the obstruction, a corresponding candidate frame from the video image stream in which the pixel of the second plurality of pixels is not present.