Display apparatus and method incorporating adaptive gaze locking
A display apparatus including: light source(s); gaze-tracking means; and processor(s) configured to: determine gaze directions of user's eyes; send, to rendering server, information indicative of gaze direction determined at first time instant; receive image frame(s) generated according to gaze, and being optionally timestamped with second time instant; display image frame(s) at third time instant; determine time lag between any one of: first time instant and third time instant, or second time instant and third time instant; detect whether or not time lag exceeds first predefined threshold; when time lag exceeds first predefined threshold, switch on gaze-lock mode; select forward line of vision as fixed gaze direction; send, to rendering server, information indicative of fixed gaze; and receive image frames generated according to fixed gaze; and display image frames.
1. A display apparatus comprising:
at least one light source per eye;
gaze-tracking means; and
at least one processor configured to:
process gaze-tracking data, collected by the gaze-tracking means, to determine gaze directions of a user's eyes;
send, to a rendering server, information indicative of a given gaze direction of a given eye determined at a first time instant (T 1 );
receive, from the rendering server, at least one image frame generated according to the given gaze direction of the given eye, the at least one image frame being timestamped with a second time instant (T 2 ) at which the rendering server generated the at least one image frame;
display the at least one image frame via the at least one light source, the at least one image frame being displayed at a third time instant (T 3 );
determine a time lag between any one of:
the first time instant (T 1 ) and the third time instant (T 3 ), or
the second time instant (T 2 ) and the third time instant (T 3 );
detect whether or not the time lag exceeds a first predefined threshold;
when it is detected that the time lag exceeds the first predefined threshold,
switch on a gaze-lock mode of the display apparatus;
select a forward line of vision of the user as a fixed gaze direction, for a given session of the gaze-lock mode;
send, to the rendering server, information indicative of the fixed gaze direction, instead of actual gaze directions of the given eye, during the given session of the gaze-lock mode; and
receive, from the rendering server, a sequence of image frames generated according to the fixed gaze direction during the given session of the gaze-lock mode; and
display the sequence of image frames via the at least one light source.
2. The display apparatus of claim 1 , wherein the at least one processor is configured to, when it is detected that the time lag does not exceed the first predefined threshold,
send, to the rendering server, information indicative of the actual gaze directions of the given eye; and
receive, from the rendering server, a sequence of image frames generated according to the actual gaze directions of the given eye.
3. The display apparatus of claim 1 , wherein the at least one processor is configured to:
detect, after the gaze-lock mode is switched on, whether or not a current time lag exceeds a second predefined threshold; and
when it is detected, after the gaze-lock mode is switched on, that the current time lag does not exceed the second predefined threshold, switch off the gaze-lock mode of the display apparatus and end the given session of the gaze-lock mode.
4. The display apparatus of claim 1 , wherein the at least one processor is configured to switch on the gaze-lock mode of the display apparatus only when it is detected that the time lag has been exceeding the first predefined threshold over a predefined time period.
5. The display apparatus of claim 1 , wherein the at least one processor is configured to employ a smoothening function to smoothen a transition from a current gaze direction of the given eye to the fixed gaze direction, wherein the smoothening function is employed over a first time period at a beginning of the given session of the gaze-lock mode.
6. The display apparatus of claim 1 , wherein the at least one processor is configured to employ a smoothening function to smoothen a transition from the fixed gaze direction to a current gaze direction of the given eye, wherein the smoothening function is employed over a second time period prior to ending the given session of the gaze-lock mode.
7. A method implemented by a display apparatus, the method comprising:
processing gaze-tracking data, collected by gaze-tracking means of the display apparatus, to determine gaze directions of a user's eyes;
sending, to a rendering server, information indicative of a given gaze direction of a given eye determined at a first time instant (T 1 );
receiving, from the rendering server, at least one image frame generated according to the given gaze direction of the given eye, the at least one image frame being timestamped with a second time instant (T 2 ) at which the rendering server generated the at least one image frame;
displaying the at least one image frame via at least one light source of the display apparatus, the at least one image frame being displayed at a third time instant (T 3 );
determining a time lag between any one of:
the first time instant (T 1 ) and the third time instant (T 3 ), or
the second time instant (T 2 ) and the third time instant (T 3 );
detecting whether or not the time lag exceeds a first predefined threshold;
when it is detected that the time lag exceeds the first predefined threshold,
switching on a gaze-lock mode of the display apparatus;
selecting a forward line of vision of the user as a fixed gaze direction, for a given session of the gaze-lock mode;
sending, to the rendering server, information indicative of the fixed gaze direction, instead of actual gaze directions of the given eye, during the given session of the gaze-lock mode; and
receiving, from the rendering server, a sequence of image frames generated according to the fixed gaze direction during the given session of the gaze-lock mode; and
displaying the sequence of image frames via the at least one light source.
8. The method of claim 7 , further comprising, when it is detected that the time lag does not exceed the first predefined threshold,
sending, to the rendering server, information indicative of the actual gaze directions of the given eye; and
receiving, from the rendering server, a sequence of image frames generated according to the actual gaze directions of the given eye.
9. The method of claim 7 , further comprising:
detecting, after the gaze-lock mode is switched on, whether or not a current time lag exceeds a second predefined threshold; and
when it is detected, after the gaze-lock mode is switched on, that the current time lag does not exceed the second predefined threshold, switching off the gaze-lock mode of the display apparatus and ending the given session of the gaze-lock mode.
10. The method of claim 7 , further comprising switching on the gaze-lock mode of the display apparatus only when it is detected that the time lag has been exceeding the first predefined threshold over a predefined time period.
11. The method of claim 7 , further comprising employing a smoothening function to smoothen a transition from a current gaze direction of the given eye to the fixed gaze direction, wherein the smoothening function is employed over a first time period at a beginning of the given session of the gaze-lock mode.
12. The method of claim 7 , further comprising employing a smoothening function to smoothen a transition from the fixed gaze direction to a current gaze direction of the given eye, wherein the smoothening function is employed over a second time period prior to ending the given session of the gaze-lock mode.
13. A rendering server configured to:
receive, from a display apparatus, information indicative of a given gaze direction of a given eye of a user determined at a first time instant (T 1 );
generate at least one image frame according to the given gaze direction of the given eye, the at least one image frame being timestamped with a second time instant (T 2 ) at which the rendering server generated the at least one image frame;
send the at least one image frame to the display apparatus for display thereat, the at least one image frame being sent at a fourth time instant (T 4 );
determine a time lag between any one of:
the first time instant (T 1 ) and the second time instant (T 2 ), or
the first time instant (T 1 ) and the fourth time instant (T 4 );
detect whether or not the time lag exceeds a third predefined threshold;
when it is detected that the time lag exceeds the third predefined threshold,
switch on a gaze-lock mode for the display apparatus;
select a forward line of vision of the user as a fixed gaze direction, for a given session of the gaze-lock mode; and
generate a sequence of image frames according to the fixed gaze direction, instead of actual gaze directions of the given eye, during the given session of the gaze-lock mode; and
send the sequence of image frames to the display apparatus for display thereat.
14. The rendering server of claim 13 , further configured to, when it is detected that the time lag does not exceed the third predefined threshold,
receive, from the display apparatus, information indicative of actual gaze directions of the given eye; and
generate a sequence of image frames according to the actual gaze directions of the given eye.
15. The rendering server of claim 13 , further configured to:
detect, after the gaze-lock mode is switched on, whether or not a current time lag exceeds a fourth predefined threshold; and
when it is detected, after the gaze-lock mode is switched on, that the current time lag does not exceed the fourth predefined threshold, switch off the gaze-lock mode for the display apparatus and end the given session of the gaze-lock mode.
16. The rendering server of claim 13 , further configured to switch on the gaze-lock mode for the display apparatus only when it is detected that the time lag has been exceeding the third predefined threshold over a predefined time period.
17. The rendering server of claim 13 , further configured to employ a smoothening function to smoothen a transition from a current gaze direction of the given eye to the fixed gaze direction, wherein the smoothening function is employed over a first time period at a beginning of the given session of the gaze-lock mode.
18. The rendering server of claim 13 , further configured to employ a smoothening function to smoothen a transition from the fixed gaze direction to a current gaze direction of the given eye, wherein the smoothening function is employed over a second time period prior to ending the given session of the gaze-lock mode.
19. A method implemented by a rendering server, the method comprising:
receiving, from a display apparatus, information indicative of a given gaze direction of a given eye of a user determined at a first time instant (T 1 );
generating at least one image frame according to the given gaze direction of the given eye, the at least one image frame being timestamped with a second time instant (T 2 ) at which the rendering server generated the at least one image frame;
sending the at least one image frame to the display apparatus for display thereat, the at least one image frame being sent at a fourth time instant (T 4 );
determining a time lag between any one of:
the first time instant (T 1 ) and the second time instant (T 2 ), or
the first time instant (T 1 ) and the fourth time instant (T 4 );
detecting whether or not the time lag exceeds a third predefined threshold;
when it is detected that the time lag exceeds the third predefined threshold,
switching on a gaze-lock mode for the display apparatus;
selecting a forward line of vision of the user as a fixed gaze direction, for a given session of the gaze-lock mode; and
generating a sequence of image frames according to the fixed gaze direction, instead of actual gaze directions of the given eye, during the given session of the gaze-lock mode; and
sending the sequence of image frames to the display apparatus for display thereat.
20. The method of claim 19 , further comprising, when it is detected that the time lag does not exceed the third predefined threshold,
receiving, from the display apparatus, information indicative of the actual gaze directions of the given eye; and
generating a sequence of image frames according to the actual gaze directions of the given eye.
21. The method of claim 19 , further comprising:
detecting, after the gaze-lock mode is switched on, whether or not a current time lag exceeds a fourth predefined threshold; and
when it is detected, after the gaze-lock mode is switched on, that the current time lag does not exceed the fourth predefined threshold, switching off the gaze-lock mode for the display apparatus and ending the given session of the gaze-lock mode.
22. The method of claim 19 , further comprising switching on the gaze-lock mode for the display apparatus only when it is detected that the time lag has been exceeding the third predefined threshold over a predefined time period.
23. The method of claim 19 , further comprising employing a smoothening function to smoothen a transition from a current gaze direction of the given eye to the fixed gaze direction, wherein the smoothening function is employed over a first time period at a beginning of the given session of the gaze-lock mode.
24. The method of claim 19 , further comprising employ a smoothening function to smoothen a transition from the fixed gaze direction to a current gaze direction of the given eye, wherein the smoothening function is employed over a second time period prior to ending the given session of the gaze-lock mode.