Imaging systems and methods for correcting visual artifacts caused by camera straylight
An imaging system for correcting visual artifacts during production of extended-reality images for display apparatus. The imaging system includes at least first camera and second camera for capturing first image and second image of real-world environment, respectively; and processor(s) configured to: analyse first and second images to identify visual artifact(s) and determine image segment of one of first image and second image that corresponds to visual artifact(s); generate image data for image segment, based on at least one of: information pertaining to virtual object, other image segment(s) neighbouring image segment, corresponding image segment in other of first image and second image, previous extended-reality image(s), photogrammetric model of real-world environment; and process one of first image and second image, based on image data, to produce extended-reality image for display apparatus.
1. An imaging system for correcting visual artifacts during production of extended-reality images for a display apparatus, the imaging system comprising:
at least a first camera and a second camera that are employed to capture contemporaneously a first image and a second image of a real-world environment, respectively; and
at least one processor configured to:
analyse the first image and the second image to identify at least one visual artifact present in one of the first image and the second image and determine an image segment of the one of the first image and the second image that corresponds to the at least one visual artifact;
divide the first image and the second image into a plurality of first sub-areas and a plurality of second sub-areas, respectively;
detect whether or not a magnitude of a difference between a luminance of a given first sub-area in the first image and a luminance of a corresponding second sub-area in the second image exceeds a first threshold difference; and
when it is detected that the magnitude of said difference exceeds the first threshold difference, determine one of the given first sub-area and the corresponding second sub-area whose luminance is higher than the luminance of other of the given first sub-area and the corresponding second sub-area, and identify the determined one of the given first sub-area and the corresponding second sub-area to have a visual artifact;
generate image data for the image segment of the one of the first image and the second image, based on at least one of: information pertaining to a virtual object that is to be virtually superimposed over at least a portion of the image segment, at least one other image segment neighbouring the image segment in the one of the first image and the second image, a corresponding image segment in other of the first image and the second image, at least one previous extended-reality image, a photogrammetric model of the real-world environment; and
process the one of the first image and the second image, based on the generated image data, to produce a given extended-reality image to be presented at the display apparatus.
2. The imaging system of claim 1 , wherein the first camera and the second camera are synchronised in at least one of: exposure time, auto exposure, auto white balance, radiometric calibration, image signal processing.
3. The imaging system of claim 1 , wherein, when analysing, the at least one processor is configured to:
divide a given image into a plurality of sub-areas;
detect whether or not a magnitude of a difference between a luminance of a given sub-area in the given image and a luminance of at least one other sub-area neighbouring the given sub-area exceeds a second threshold difference; and
when it is detected that the magnitude of said difference exceeds the second threshold difference, determine one of the given sub-area and the at least one other sub-area whose luminance is higher than the luminance of other of the given sub-area and the at least one other sub-area, and identify the determined one of the given sub-area and the at least one other sub-area to have a visual artifact.
4. The imaging system of claim 1 , wherein the at least one processor is configured to analyse the first image and the second image to identify the at least one visual artifact, based on information pertaining to at least one light source present in the real-world environment, said information comprising at least one of: a location of the at least one light source in the real-world environment, a direction in which light emanating from the at least one light source is directed within the real-world environment, an intensity of the light emanating from the at least one light source, a wavelength of the light emanating from the at least one light source, a shape of the at least one light source.
5. The imaging system of claim 1 , wherein a given image is captured using a default exposure setting of a given camera, and wherein the given camera is to be employed to capture at least one underexposed image using an adjusted exposure setting of the given camera, wherein, when analysing, the at least one processor is configured to:
divide the given image and the at least one underexposed image into a plurality of sub-areas and a plurality of underexposed sub-areas, respectively;
detect whether or not a magnitude of a difference between a luminance of a given sub-area in the given image and a luminance of a corresponding underexposed sub-area in the at least one underexposed image exceeds a third threshold difference; and
when it is detected that the magnitude of said difference exceeds the third threshold difference, identify the given sub-area of the given image to have a visual artifact.
6. The imaging system of claim 1 , wherein, when analysing, the at least one processor is configured to determine a shape of the at least one visual artifact based on a shape of an aperture of a given camera.
7. The imaging system of claim 1 , further comprising pose-tracking means, a given camera being employed to capture a plurality of consecutive images, wherein the at least one processor is configured to:
process pose-tracking data, collected by the pose-tracking means, to track a pose of the given camera;
determine a first motion vector representing a movement of the given camera, based on the tracked pose of the given camera;
determine a second motion vector representing a change in a position of a given visual artifact across the plurality of consecutive images;
detect whether or not an angle between the first motion vector and the second motion vector exceeds a threshold angle; and
when it is detected that the angle between the first motion vector and the second motion vector exceeds the threshold angle, discard the given visual artifact.
8. The imaging system of claim 7 , wherein the at least one processor is configured to:
detect whether or not a shape of the given visual artifact changes across the plurality of consecutive images; and
when it is detected that the angle between the first motion vector and the second motion vector does not exceed the threshold angle, but the shape of the given visual artifact changes across the plurality of consecutive images, discard the given visual artifact.
9. The imaging system of claim 1 , wherein the at least one processor is configured to:
access, from a data repository, a lighting model of the real-world environment;
detect whether or not a magnitude of a difference between a luminance of the first image and a luminance of the second image exceeds a fourth threshold difference; and
when it is detected that the magnitude of said difference exceeds the fourth threshold difference, determine, based on the lighting model of the real-world environment, if any of the first camera and the second camera has moisture or dirt.
10. The imaging system of claim 1 , wherein, when analysing, the at least one processor is configured to:
divide the first image and the second image into a plurality of first sub-areas and a plurality of second sub-areas, respectively;
detect whether or not a magnitude of a difference between a colour value of a given first sub-area in the first image and a colour value of a corresponding second sub-area in the second image exceeds a threshold colour difference; and
when it is detected that the magnitude of said difference exceeds the threshold colour difference, determine one of the given first sub-area and the corresponding second sub-area whose colour value is higher than the colour value of other of the given first sub-area and the corresponding second sub-area, and identify the determined one of the given first sub-area and the corresponding second sub-area to have a visual artifact.
11. A method for correcting visual artifacts during production of extended-reality images for a display apparatus, the method comprising:
capturing contemporaneously, via at least a first camera and a second camera, a first image and a second image of a real-world environment, respectively;
analysing the first image and the second image to identify at least one visual artifact present in one of the first image and the second image and determine an image segment of the one of the first image and the second image that corresponds to the at least one visual artifact;
dividing the first image and the second image into a plurality of first sub-areas and a plurality of second sub-areas, respectively;
detecting whether or not a magnitude of a difference between a luminance of a given first sub-area in the first image and a luminance of a corresponding second sub-area in the second image exceeds a first threshold difference; and
when it is detected that the magnitude of said difference exceeds the first threshold difference, determining one of the given first sub-area and the corresponding second sub-area whose luminance is higher than the luminance of other of the given first sub-area and the corresponding second sub-area, and identifying the determined one of the given first sub-area and the corresponding second sub-area to have a visual artifact;
generating image data for the image segment of the one of the first image and the second image, based on at least one of: information pertaining to a virtual object that is to be virtually superimposed over at least a portion of the image segment, at least one other image segment neighbouring the image segment in the one of the first image and the second image, a corresponding image segment of other of the first image and the second image, at least one previous extended-reality image, a photogrammetric model of the real-world environment; and
processing the one of the first image and the second image, based on the generated image data, to produce a given extended-reality image to be presented at the display apparatus.
12. The method of claim 11 , further comprising synchronising the first camera and the second camera in at least one of: exposure time, auto exposure, auto white balance, radiometric calibration, image signal processing.
13. The method of claim 11 , wherein the step of analysing comprises:
dividing a given image into a plurality of sub-areas;
detecting whether or not a magnitude of a difference between a luminance of a given sub-area in the given image and a luminance of at least one other sub-area neighbouring the given sub-area exceeds a second threshold difference; and
when it is detected that the magnitude of said difference exceeds the second threshold difference, determining one of the given sub-area and the at least one other sub-area whose luminance is higher than the luminance of other of the given sub-area and the at least one other sub-area, and identifying the determined one of the given sub-area and the at least one other sub-area to have a visual artifact.
14. The method of claim 11 , wherein the step of analysing is performed based on information pertaining to at least one light source present in the real-world environment, said information comprising at least one of: a location of the at least one light source in the real-world environment, a direction in which light emanating from the at least one light source is directed within the real-world environment, an intensity of the light emanating from the at least one light source, a wavelength of the light emanating from the at least one light source, a shape of the at least one light source.
15. The method of claim 11 , wherein a given image is captured using a default exposure setting of a given camera, and wherein the given camera is employed to capture at least one underexposed image using an adjusted exposure setting of the given camera, wherein the step of analysing comprises:
dividing the given image and the at least one underexposed image into a plurality of sub-areas and a plurality of underexposed sub-areas, respectively;
detecting whether or not a magnitude of a difference between a luminance of a given sub-area in the given image and a luminance of a corresponding underexposed sub-area in the at least one underexposed image exceeds a third threshold difference; and
when it is detected that the magnitude of said difference exceeds the third threshold difference, identifying the given sub-area of the given image to have a visual artifact.
16. The method of claim 11 , wherein the step of analysing comprises determining a shape of the at least one visual artifact based on a shape of an aperture of a given camera.
17. The method of claim 11 , wherein a given camera is employed to capture a plurality of consecutive images, the method further comprising:
processing pose-tracking data, collected by pose-tracking means, to track a pose of the given camera;
determining a first motion vector representing a movement of the given camera, based on the tracked pose of the given camera;
determining a second motion vector representing a change in a position of a given visual artifact across the plurality of consecutive images;
detecting whether or not an angle between the first motion vector and the second motion vector exceeds a threshold angle; and
when it is detected that the angle between the first motion vector and the second motion vector exceeds the threshold angle, discarding the given visual artifact.
18. The method of claim 17 , further comprising:
detecting whether or not a shape of the given visual artifact changes across the plurality of consecutive images; and
when it is detected that the angle between the first motion vector and the second motion vector does not exceed the threshold angle, but the shape of the given visual artifact changes across the plurality of consecutive images, discarding the given visual artifact.
19. The method of claim 11 , further comprising:
accessing, from a data repository, a lighting model of the real-world environment;
detecting whether or not a magnitude of a difference between a luminance of the first image and a luminance of the second image exceeds a fourth threshold difference; and
when it is detected that the magnitude of said difference exceeds the fourth threshold difference, determining, based on the lighting model of the real-world environment, if any of the first camera and the second camera has moisture or dirt.
20. The method of claim 11 , wherein the step of analysing comprises:
dividing the first image and the second image into a plurality of first sub-areas and a plurality of second sub-areas, respectively;
detecting whether or not a magnitude of a difference between a colour value of a given first sub-area in the first image and a colour value of a corresponding second sub-area in the second image exceeds a threshold colour difference; and
when it is detected that the magnitude of said difference exceeds the threshold colour difference, determining one of the given first sub-area and the corresponding second sub-area whose colour value is higher than the colour value of other of the given first sub-area and the corresponding second sub-area, and identifying the determined one of the given first sub-area and the corresponding second sub-area to have a visual artifact.