Detection and mitigation of defects in multiscopic displays

1 . A system comprising:

a tracker;

a multiscopic display;

at least one camera that is positioned such that a display area of the multiscopic display lies within a field of view of the at least one camera at least partially; and

at least one processor configured to:

perform calibration to determine at least one parameter of a multiscopic optical element of the multiscopic display;

determine respective optical locations of eyes of at least one user relative to the display area, using the tracker;

determine or retrieve an optical location of the at least one camera relative to the display area;

generate or retrieve respective images to be directed toward the eyes of the at least one user, based on the respective optical locations of the eyes relative to the display area;

generate a first multiscopic image to be displayed via the multiscopic display, based on the respective images for the eyes of the at least one user, and the at least one parameter of the multiscopic optical element;

display the first multiscopic image via the multiscopic display, whilst capturing at least one image of the display area using the at least one camera;

calculate at least one expected image of the display area that is expected to be captured by the at least one camera during display of the first multiscopic image, based on the first multiscopic image, the at least one parameter of the multiscopic optical element, the respective optical locations of the eyes relative to the display area, and the optical location of the at least one camera relative to the display area;

identify at least one defective sub-region of the display area, based on a comparison between the at least one expected image and the at least one captured image of the display area;

determine whether the at least one defective sub-region overlaps at least partially with a sub-region of the display area to be used to present at least one virtual object, based on a location of the at least one virtual object in a second multiscopic image; and

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

identify another sub-region of the display area that does not overlap with the at least one defective sub-region;

relocate the at least one virtual object within the second multiscopic image to use the another sub-region for presenting the at least one virtual object; and

display the second multiscopic image after relocating the at least one virtual object.

2 . The system of claim 1 , wherein the at least one processor is configured to:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

determine whether another sub-region of the display area that does not overlap with the at least one defective sub-region cannot be identified, or whether the at least one virtual object is not to be relocated; and

when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determine colours and intensities of pixels in a portion of the second multiscopic image to be displayed via the at least one defective sub-region; and

adjust the colours and intensities of said pixels, such that deviations between the adjusted colours and intensities and original colours and intensities of said pixels lie within a predefined threshold deviation.

3 . The system of claim 1 , wherein the at least one processor is configured to:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

determine whether another sub-region of the display area that does not overlap with the at least one defective sub-region cannot be identified, or whether the at least one virtual object is not to be relocated; and

when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determine gaze directions of the eyes of the at least one user, using the tracker;

determine which eye of the at least one user is focused on the at least one defective sub-region, based on the gaze directions; and

adjust colours and intensities of pixels in a portion of the second multiscopic image to be displayed via the at least one defective sub-region, based on colours and intensities of pixels in a corresponding portion of an image to be directed toward the determined eye, the corresponding portion presenting the at least one virtual object.

4 . The system of claim 3 , wherein the at least one processor is configured to:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object, and when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determine whether both the eyes of the at least one user are focused on the at least one defective sub-region; and

when it is determined that both the eyes are focused on the at least one defective sub-region,

determine a dominant eye of the at least one user; and

adjust the colours and intensities of the pixels in the portion of the second multiscopic image to be displayed via the at least one defective sub-region, based on colours and intensities of pixels in a corresponding portion of an image to be directed toward the dominant eye, the corresponding portion presenting the at least one virtual object.

5 . The system of claim 3 , wherein the at least one processor is configured to:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object, and when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determine a portion of another image to be directed toward another eye of the at least one user, said portion presenting the at least one virtual object and being displayed via a non-defective sub-region of the display area; and

increase intensities of pixels in said portion of the another image, prior to generating the second multiscopic image from the image to be directed toward the determined eye and the another image to be directed toward the another eye.

6 . The system of claim 5 , wherein the at least one processor is configured to decrease the intensities of the pixels in the corresponding portion of the image to be directed toward the determined eye, prior to generating the second multiscopic image from the image to be directed toward the determined eye and the another image to be directed toward the another eye.

7 . The system of claim 6 , wherein the intensities of the pixels in the corresponding portion of the image are decreased and the intensities of the pixels in said portion of the another image are increased, such that a binocular sum of:

(i) a luminance of light emitted from the at least one defective sub-region toward the determined eye, and

(ii) a luminance of light emitted from the non-defective sub-region toward the another eye,

lies within a predefined threshold range of a target binocular luminance corresponding to a part of the at least one virtual object that is being presented via the at least one defective sub-region.

8 . The system of claim 1 , wherein the at least one processor is configured to:

determine sub-pixels of the second multiscopic image whose light is not directed toward any of the eyes of the at least one user through a sub-region of the display area that surrounds the at least one defective sub-region, based on the at least one parameter of the multiscopic optical element and the respective optical locations of the eyes of the at least one user relative to the display area; and

adjust colours and intensities of the determined sub-pixels, based on colours and intensities of pixels in a portion of the second multiscopic image to be displayed via the at least one defective sub-region.

9 . The system of claim 1 , wherein the at least one processor is configured to:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

determine whether a virtual depth at which the at least one virtual object is to be presented lies within a predefined threshold range from a native optical distance of the multiscopic display; and

when it is determined that the virtual depth lies within the predefined threshold range from the native optical distance, generate the second multiscopic image to present the at least one virtual object at the native optical distance.

10 . A method comprising:

performing calibration to determine at least one parameter of a multiscopic optical element of a multiscopic display;

determining respective optical locations of eyes of at least one user relative to a display area of the multiscopic display, using a tracker;

determining or retrieving an optical location of at least one camera relative to the display area, wherein the at least one camera is positioned such that the display area of the multiscopic display lies within a field of view of the at least one camera at least partially;

generating or retrieving respective images to be directed toward the eyes of the at least one user, based on the respective optical locations of the eyes relative to the display area;

generating a first multiscopic image to be displayed via the multiscopic display, based on the respective images for the eyes of the at least one user, and the at least one parameter of the multiscopic optical element;

displaying the first multiscopic image via the multiscopic display, whilst capturing at least one image of the display area using the at least one camera;

calculating at least one expected image of the display area that is expected to be captured by the at least one camera during display of the first multiscopic image, based on the first multiscopic image, the at least one parameter of the multiscopic optical element, the respective optical locations of the eyes relative to the display area, and the optical location of the at least one camera relative to the display area;

identifying at least one defective sub-region of the display area, based on a comparison between the at least one expected image and the at least one captured image of the display area;

determining whether the at least one defective sub-region overlaps at least partially with a sub-region of the display area to be used to present at least one virtual object, based on a location of the at least one virtual object in a second multiscopic image; and

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

identifying another sub-region of the display area that does not overlap with the at least one defective sub-region;

relocating the at least one virtual object within the second multiscopic image to use the another sub-region for presenting the at least one virtual object; and

displaying the second multiscopic image after relocating the at least one virtual object.

11 . The method of claim 10 , further comprising:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

determining whether another sub-region of the display area that does not overlap with the at least one defective sub-region cannot be identified, or whether the at least one virtual object is not to be relocated; and

when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determining colours and intensities of pixels in a portion of the second multiscopic image to be displayed via the at least one defective sub-region; and

adjusting the colours and intensities of said pixels, such that deviations between the adjusted colours and intensities and original colours and intensities of said pixels lie within a predefined threshold deviation.

12 . The method of claim 10 , further comprising:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

determining whether another sub-region of the display area that does not overlap with the at least one defective sub-region cannot be identified, or whether the at least one virtual object is not to be relocated; and

when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determining gaze directions of the eyes of the at least one user, using the tracker;

determining which eye of the at least one user is focused on the at least one defective sub-region, based on the gaze directions; and

adjusting colours and intensities of pixels in a portion of the second multiscopic image to be displayed via the at least one defective sub-region, based on colours and intensities of pixels in a corresponding portion of an image to be directed toward the determined eye, the corresponding portion presenting the at least one virtual object.

13 . The method of claim 12 , further comprising:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object, and when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determining whether both the eyes of the at least one user are focused on the at least one defective sub-region; and

when it is determined that both the eyes are focused on the at least one defective sub-region,

determining a dominant eye of the at least one user; and

adjusting the colours and intensities of the pixels in the portion of the second multiscopic image to be displayed via the at least one defective sub-region, based on colours and intensities of pixels in a corresponding portion of an image to be directed toward the dominant eye, the corresponding portion presenting the at least one virtual object.

14 . The method of claim 12 , further comprising:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object, and when it is determined that another sub-region of the display area cannot be identified, or the at least one virtual object is not to be relocated,

determining a portion of another image to be directed toward another eye of the at least one user, said portion presenting the at least one virtual object and being displayed via a non-defective sub-region of the display area; and

increasing intensities of pixels in said portion of the another image, prior to generating the second multiscopic image from the image to be directed toward the determined eye and the another image to be directed toward the another eye.

15 . The method of claim 14 , further comprising decreasing the intensities of the pixels in the corresponding portion of the image to be directed toward the determined eye, prior to generating the second multiscopic image from the image to be directed toward the determined eye and the another image to be directed toward the another eye.

16 . The method of claim 15 , the intensities of the pixels in the corresponding portion of the image are decreased and the intensities of the pixels in the portion of the another image are increased, such that a binocular sum of:

(i) a luminance of light emitted from the at least one defective sub-region toward the determined eye, and

(ii) a luminance of light emitted from the non-defective sub-region toward the another eye,

lies within a predefined threshold range of a target binocular luminance corresponding to a part of the at least one virtual object that is being presented via the at least one defective sub-region.

17 . The method of claim 10 , further comprising:

determining sub-pixels of the second multiscopic image whose light is not directed toward any of the eyes of the at least one user through a sub-region of the display area that surrounds the at least one defective sub-region, based on the at least one parameter of the multiscopic optical element and the respective optical locations of the eyes of the at least one user relative to the display area; and

adjusting colours and intensities of the determined sub-pixels, based on colours and intensities of pixels in a portion of the second multiscopic image to be displayed via the at least one defective sub-region.

18 . The method of claim 10 , further comprising:

when it is determined that the at least one defective sub-region overlaps at least partially with the sub-region to be used to present the at least one virtual object,

determining whether a virtual depth at which the at least one virtual object is to be presented lies within a predefined threshold range from a native optical distance of the multiscopic display; and

when it is determined that the virtual depth lies within the predefined threshold range from the native optical distance, generating the second multiscopic image to present the at least one virtual object at the native optical distance.