Hybrid heads-up display for presenting virtual content outside boundaries

1 . A system comprising:

tracking means;

a light field display unit;

an optical combiner; and

at least one processor configured to:

determine a relative location of a first eye and a second eye of at least one user with respect to the optical combiner, by utilising the tracking means;

determine a position and an orientation of at least a first region of a light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, by utilising the tracking means;

generate a first region of a given light field image to be displayed via the light field display unit, based on the position and the orientation of at least the first region of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, wherein the first region of the light-emitting component is employed to display the first region of the given light field image;

generate a second region of the given light field image, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and a position and an orientation of a second region of the light-emitting component of the light field display unit with respect to the optical combiner, wherein the second region of the light-emitting component is employed to display the second region of the given light field image, the second region being different from the first region; and

display the given light field image via the light field display unit to produce a synthetic light field, wherein a first part of the synthetic light field corresponding to the first region of the given light field image is directed towards the at least one user, while a second part of the synthetic light field corresponding to the second region of the given light field image is directed towards the optical combiner,

wherein the optical combiner is employed to reflect respective sub-parts of the second part of the synthetic light field towards the first eye and the second eye of the at least one user, while optically combining a real-world light field with the respective sub-parts of the second part of the synthetic light field.

2 . The system of claim 1 , wherein the light field display unit further comprises an active optical element arranged on an optical path of the light-emitting component, wherein the at least one processor is configured to:

control a first region of the active optical element that corresponds to the first region of the light-emitting component of the light field display unit, based on the position and the orientation of the first region of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective sub-parts of the first part of the synthetic light field corresponding to the first region of the given light field image towards the first eye and the second eye of the at least one user.

3 . The system of claim 1 , wherein the light field display unit further comprises an active optical element arranged on an optical path of the light-emitting component, wherein the at least one processor is configured to:

control a first region of the active optical element that corresponds to the first region of the light-emitting component of the light field display unit, based on the position and the orientation of the first region of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective sub-parts of the first part of the synthetic light field corresponding to the first region of the given light field image towards the first eye and the second eye of the at least one user; and

control a second region of the active optical element that corresponds to the second region of the light-emitting component of the light field display unit, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and the position and the orientation of the second region of the light-emitting component with respect to the optical combiner, to re-direct the respective sub-parts of the second part of the synthetic light field corresponding to the second region of the given light field image towards the optical combiner.

4 . The system of claim 1 , wherein the light field display unit further comprises an active optical element arranged on an optical path of the light-emitting component, wherein the at least one processor is configured to:

detect when virtual content is to be presented directly via the light field display unit only; and

when it is detected that the virtual content is to be presented directly via the light field display unit only,

generate another light field image to be displayed via the light field display unit, based on a position and an orientation of at least a region of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, wherein at least said region of the light-emitting component is to be employed to present the virtual content;

display the another light field image via the light field display unit to produce another synthetic light field; and

control the active optical element, based on the position and the orientation of at least said region of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective parts of the another synthetic light field towards the first eye and the second eye of the at least one user.

5 . The system of claim 1 , wherein the light field display unit further comprises an active optical element arranged on an optical path of the light-emitting component, wherein the at least one processor is configured to:

detect when virtual content is to be presented via the optical combiner only; and

when it is detected that the virtual content is to be presented via the optical combiner only,

generate yet another light field image to be displayed via the light field display unit, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and a position and an orientation of at least a region of the light-emitting component of the light field display unit with respect to the optical combiner, wherein at least said region of the light-emitting component is to be employed to present the virtual content;

display the yet another light field image via the light field display unit to produce yet another synthetic light field; and

control the active optical element, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and the position and the orientation of at least said region of the light-emitting component with respect to the optical combiner, to re-direct the yet another synthetic light field towards the optical combiner,

wherein the optical combiner is employed to reflect respective parts of the yet another synthetic light field towards the first eye and the second eye of the at least one user, while optically combining the real-world light field with the respective parts of the yet another synthetic light field.

6 . The system of claim 1 , wherein the light field display unit further comprises an active optical element arranged on an optical path of the light-emitting component, wherein the at least one processor is configured to:

generate a first light field image to be displayed via the light field display unit, based on a position and an orientation of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user;

generate a second light field image to be displayed via the light field display unit, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and a position and an orientation of the light-emitting component of the light field display unit with respect to the optical combiner;

display the first light field image and the second light field image, via the light field display unit, to produce a first synthetic light field and a second synthetic light field, respectively, wherein the first light field image and the second light field image are displayed by employing temporal multiplexing;

when displaying the first light field image, control the active optical element, based on the position and the orientation of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective parts of the first synthetic light field towards the first eye and the second eye of the at least one user; and

when displaying the second light field image, control the active optical element, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and the position and the orientation of the light-emitting component with respect to the optical combiner, to re-direct the second synthetic light field towards the optical combiner, wherein the optical combiner is employed to reflect respective parts of the second synthetic light field towards the first eye and the second eye of the at least one user, while optically combining the real-world light field with the respective parts of the second synthetic light field.

7 . The system of claim 1 , wherein at least a first sub-region of the first region of the given light field image and at least a second sub-region of the second region of the given light field image are generated based on same virtual content, wherein the first sub-region of the first region is adjacent to the second sub-region of the second region.

8 . The system of claim 1 , further comprising at least one real-world facing camera, wherein the at least one processor is configured to:

detect when at least one virtual object is to be presented simultaneously via the optical combiner as well as directly via the light field display unit; and

when it is detected that the at least one virtual object is to be presented simultaneously via the optical combiner as well as directly via the light field display unit,

capture at least one real-world image of a region of a real-world environment that lies in a field of view of the at least one user when viewing through the optical combiner, by utilising the at least one real-world facing camera; and

generate a first synthetic image and a second synthetic image by utilising the at least one real-world image, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and the position and the orientation of at least the first region of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, respectively,

wherein when generating the first region of the given light field image, the at least one processor is configured to generate at least a first sub-region of the first region of the given light field image by utilising the first synthetic image and the second synthetic image.

9 . The system of claim 8 , wherein the at least one processor is configured to determine an amount of attenuation caused by the optical combiner to the real-world light field passing therethrough,

wherein when generating the first region of the given light field image, the at least one processor is configured to generate intensity values of at least the first sub-region of the first region, based on the amount of attenuation.

10 . The system of claim 1 , wherein the at least one processor is configured to determine an average intensity of a real-world light field passing through the optical combiner,

wherein when generating the first region of the given light field image, the at least one processor is configured to generate intensity values of at least a first sub-region of the first region, based on the average intensity of the real-world light field.

11 . The system of claim 1 , wherein the system is implemented in a vehicle, the system further comprising a set of cameras arranged on an external body of the vehicle, wherein the at least one processor is configured to:

capture a set of real-world images of a region of a real-world environment that surrounds the vehicle, by utilising the set of cameras; and

generate at least one of: (i) a view of said region of the real-world environment, (ii) a 360-degree view of the vehicle in said region of the real-world environment, by utilising the set of real-world images,

wherein when generating the first region of the given light field image, the at least one processor is configured to generate at least a background of the first region by utilising the at least one of: (i) the view of said region of the real-world environment, (ii) the 360-degree view of the vehicle in said region of the real-world environment.

12 . The system of claim 1 , wherein the system is implemented in a vehicle, the system further comprising lane detectors, wherein the at least one processor is configured to determine a relative position of the vehicle with respect to lane markings on a road,

wherein when generating the first region of the given light field image, the at least one processor is configured to:

depict external boundaries of the vehicle in the first region of the given light field image by utilising a 3D model of the vehicle, based on the position and the orientation of at least the first region of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, and a position and an orientation of the first region of the light-emitting component in the vehicle; and

depict, in the first region of the given light field image, the lane markings with respect to the external boundaries of the vehicle, based on the relative position of the vehicle with respect to the lane markings.

13 . A system comprising:

tracking means;

a light field display unit;

an optical combiner; and

at least one processor configured to:

determine a relative location of a first eye and a second eye of at least one user with respect to the optical combiner, by utilising the tracking means;

determine a position and an orientation of at least a first region of a light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, by utilising the tracking means;

generate a first region of a given light field image to be displayed via the light field display unit, based on the position and the orientation of at least the first region of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, wherein the first region of the light-emitting component is to be employed to display the first region of the given light field image;

generate a second region of the given light field image, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and a position and an orientation of a second region of the light-emitting component of the light field display unit with respect to the optical combiner, wherein the second region of the light-emitting component is to be employed to display the second region of the given light field image, the second region being different from the first region; and

display the given light field image via the light field display unit to produce a synthetic light field, wherein a first part of the synthetic light field corresponding to the first region of the given light field image is directed towards the at least one user, while a second part of the synthetic light field corresponding to the second region of the given light field image is directed towards the optical combiner,

wherein the optical combiner is employed to reflect respective sub-parts of the second part of the synthetic light field towards the first eye and the second eye of the at least one user, while optically combining a real-world light field with the respective sub-parts of the second part of the synthetic light field, wherein the at least one processor is further configured to:

determine gaze directions of eyes of the at least one user, by utilising the tracking means;

detect whether the gaze directions of the eyes of the at least one user pass through the light field display unit or the optical combiner; and

when it is detected that the gaze directions pass through the optical combiner, perform at least one of:

lower a resolution of respective first regions of light field images to be displayed;

selectively lower a frame rate at which the respective first regions of the light field images are to be displayed;

generate the respective first regions of the light field images to present at least one virtual object at an optical depth that is within a predefined threshold distance from a distance between the light-emitting component of the light field display unit and the at least one user.

14 . A method comprising:

determining a relative location of a first eye and a second eye of at least one user with respect to an optical combiner, by utilising tracking means;

determining a position and an orientation of at least a first region of a light-emitting component of a light field display unit with respect to the first eye and the second eye of the at least one user, by utilising the tracking means;

generating a first region of a given light field image to be displayed via the light field display unit, based on the position and the orientation of at least the first region of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, wherein the first region of the light-emitting component is to be employed to display the first region of the given light field image;

generating a second region of the given light field image, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and a position and an orientation of a second region of the light-emitting component of the light field display unit with respect to the optical combiner, wherein the second region of the light-emitting component is to be employed to display the second region of the given light field image, the second region being different from the first region; and

displaying the given light field image via the light field display unit to produce a synthetic light field, wherein a first part of the synthetic light field corresponding to the first region of the given light field image is directed towards the at least one user, while a second part of the synthetic light field corresponding to the second region of the given light field image is directed towards the optical combiner,

wherein the optical combiner is employed to reflect respective sub-parts of the second part of the synthetic light field towards the first eye and the second eye of the at least one user, while optically combining a real-world light field with the respective sub-parts of the second part of the synthetic light field.

15 . The method of claim 14 , further comprising controlling a first region of an active optical element that corresponds to the first region of the light-emitting component of the light field display unit, based on the position and the orientation of the first region of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective sub-parts of the first part of the synthetic light field corresponding to the first region of the given light field image towards the first eye and the second eye of the at least one user, wherein the active optical element is arranged on an optical path of the light-emitting component.

16 . The method of claim 14 , further comprising:

controlling a first region of an active optical element that corresponds to the first region of the light-emitting component of the light field display unit, based on the position and the orientation of the first region of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective sub-parts of the first part of the synthetic light field corresponding to the first region of the given light field image towards the first eye and the second eye of the at least one user, wherein the active optical element is arranged on an optical path of the light-emitting component; and

controlling a second region of the active optical element that corresponds to the second region of the light-emitting component of the light field display unit, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and the position and the orientation of the second region of the light-emitting component with respect to the optical combiner, to re-direct the respective sub-parts of the second part of the synthetic light field corresponding to the second region of the given light field image towards the optical combiner.

17 . The method of claim 14 , further comprising:

detecting when virtual content is to be presented directly via the light field display unit only; and

when it is detected that the virtual content is to be presented directly via the light field display unit only,

generating another light field image to be displayed via the light field display unit, based on a position and an orientation of at least a region of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user, wherein at least said region of the light-emitting component is to be employed to present the virtual content;

displaying the another light field image via the light field display unit to produce another synthetic light field; and

controlling an active optical element, based on the position and the orientation of at least said region of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective parts of the another synthetic light field towards the first eye and the second eye of the at least one user, wherein the active optical element is arranged on an optical path of the light-emitting component.

18 . The method of claim 14 , further comprising:

detecting when virtual content is to be presented via the optical combiner only; and

when it is detected that the virtual content is to be presented via the optical combiner only,

generating yet another light field image to be displayed via the light field display unit, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and a position and an orientation of at least a region of the light-emitting component of the light field display unit with respect to the optical combiner, wherein at least said region of the light-emitting component is to be employed to present the virtual content;

displaying the yet another light field image via the light field display unit to produce yet another synthetic light field; and

controlling an active optical element, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and the position and the orientation of at least said region of the light-emitting component with respect to the optical combiner, to re-direct the yet another synthetic light field towards the optical combiner, wherein the active optical element is arranged on an optical path of the light-emitting component,

wherein the optical combiner is employed to reflect respective parts of the yet another synthetic light field towards the first eye and the second eye of the at least one user, while optically combining the real-world light field with the respective parts of the yet another synthetic light field.

19 . The method of claim 14 , further comprising:

generating a first light field image to be displayed via the light field display unit, based on a position and an orientation of the light-emitting component of the light field display unit with respect to the first eye and the second eye of the at least one user;

generating a second light field image to be displayed via the light field display unit, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and a position and an orientation of the light-emitting component of the light field display unit with respect to the optical combiner;

displaying the first light field image and the second light field image, via the light field display unit, to produce a first synthetic light field and a second synthetic light field, respectively, wherein the first light field image and the second light field image are displayed by employing temporal multiplexing;

when displaying the first light field image, controlling an active optical element, based on the position and the orientation of the light-emitting component with respect to the first eye and the second eye of the at least one user, to re-direct respective parts of the first synthetic light field towards the first eye and the second eye of the at least one user, wherein the active optical element is arranged on an optical path of the light-emitting component; and

when displaying the second light field image, controlling the active optical element, based on the relative location of the first eye and the second eye of the at least one user with respect to the optical combiner, and the position and the orientation of the light-emitting component with respect to the optical combiner, to re-direct the second synthetic light field towards the optical combiner, wherein the optical combiner is employed to reflect respective parts of the second synthetic light field towards the first eye and the second eye of the at least one user, while optically combining the real-world light field with the respective parts of the second synthetic light field.

20 . The method of claim 14 , wherein at least a first sub-region of the first region of the given light field image and at least a second sub-region of the second region of the given light field image are generated based on same virtual content, wherein the first sub-region of the first region is adjacent to the second sub-region of the second region.