Correlating Pupil Position to Gaze Location Within a Scene

1 . A method comprising:

receiving, by a first computer system, a first video stream depicting an eye of a user, the first video stream comprising a first plurality of frames;

receiving, by the first computer system, a second video stream depicting a scene in front of the user, the second video stream comprising a second plurality of frames;

determining, by the computer system, pupil position within the first plurality of frames;

calculating, by the first computer system, gaze location in the second plurality of frames based on pupil position within the first plurality of frames; and

sending an indication of the gaze location to a second computer system, the second computer system distinct from the first computer system, and the sending in real-time with creation of the first video stream.

2 . The method of claim 1 further comprising, prior to calculating and sending:

calibrating a relationship between pupil position within the first plurality of frames and gaze location in the second plurality of frames, the calibrating by:

displaying a plurality of calibration features in the scene in front of the user;

determining, by the first computer system, location of the calibration features within the second plurality of frames;

relating, by the first computer system, pupil position in the first plurality of frames to location of the calibration features in the second plurality of frames; and thereby

creating, by the first computer system, a homography that relates pupil position in the first plurality of frames to gaze location in the second plurality of frames.

3 . The method of claim 2 wherein displaying the calibration features further comprises revealing the plurality of calibration features sequentially, the revealing of the plurality of calibration features not on a computer monitor.

4 . The method of claim 2 wherein displaying the calibration features further comprises sequentially revealing the plurality of calibration features on a display device of a computer system.

5 . The method of claim 2 wherein relating further comprises:

clustering, by the first computer system, indications of pupil position derived from the first plurality of frames, the clustering creates a plurality of pupil clusters;

clustering, by the second computer system, indications of location of the calibration features in the second plurality of frames, the clustering indications of location creates a plurality of feature clusters; and

correlating the pupil clusters to the feature clusters.

6 . The method of claim 5 wherein, prior to correlating, discarding at least one cluster from the plurality of pupil clusters.

7 . The method of claim 5 wherein correlating further comprises associating appearance of calibration features in the scene with pupil clusters based on frame numbers associated with each pupil position determination.

8 . The method of claim 1 further comprising:

identifying, by the first computer system, a computer monitor visible in the second plurality of frames;

sending an indication of the location of the computer monitor within the second video stream, the sending to the second computer system; and

sending an indication of a resolution of the second video stream, the sending to the second computer system.

9 . The method of claim 8 wherein identifying further comprises at least one selected from the group consisting of: identifying a border of predetermined color shown on the computer monitor; identifying four indicia of corner locations shown on the computer monitor; and identifying a border of predetermined color that at least partially circumscribes the computer monitor.

10 . The method of claim 8 further comprising:

receiving, by the second computer system, the indication of gaze location, the indication of location of the display device within the second video stream, and the indication of resolution of the second video stream;

performing, by the second computer system, a perspective transformation resulting in a desired location on the display device; and

moving, by the second computer system, a cursor shown on the computer monitor to the desired location.

11 . A computer system comprising:

a processor;

a memory coupled to the processor; and

a wireless communication subsystem coupled to the processor;

wherein the memory storing a program that, when executed by the processor, causes the processor to:

receive a first video stream depicting an eye of a user, the first video stream comprising a first plurality of frames;

receive a second video stream depicting a scene in front of the user, the second video stream comprising a second plurality of frames;

determine pupil position within the first plurality of frames;

calculate gaze location in the second plurality of frames based on pupil position within the first plurality of frames; and

send, by way of the wireless communication subsystem, an indication of the gaze location to a remote computer system, and the send in real-time with receipt of the first video stream.

12 . The computer system of claim 11 wherein the program further causes the processor to:

calibrate a relationship between pupil position within the first plurality of frames and gaze location in the second plurality of frames, the calibration by causing the processor to:

determine location of a plurality of calibration features within the second plurality of frames;

relate pupil position in the first plurality of frames to location of the calibration features in the second plurality of frames; and thereby

create a homography that relates pupil position in the first plurality of frames to gaze location in the second plurality of frames.

13 . The computer system of claim 12 wherein as part of calibration, the program further causes the processor to cause a display device of the remote computer system to display the plurality of calibration features.

14 . The computer system of claim 12 wherein when the processor relates pupil position to the location of the calibration features, the program causes the processor to:

cluster indications of pupil position derived from the first plurality of frames, the clustering creates a plurality of pupil clusters;

cluster indications of location of the calibration features in the second plurality of frames to create a plurality of feature clusters; and

correlate the pupil clusters to the feature clusters.

15 . The computer system of claim 14 wherein the program further causes the processor, prior to the correlation, to discard at least one cluster from the plurality of pupil clusters.

16 . The computer system of claim 14 wherein when the processor correlates, the program causes the processor to associate appearance of calibration features in the scene with pupil clusters based on frame numbers associated with each pupil position determination.

17 . The computer system of claim 11 wherein the program further causes the processor to:

identify a display device visible in the second plurality of frames;

send an indication of the location of the display device within the second video stream, the sending to the second computer system; and

send an indication of a resolution of the second video stream, the sending to the second computer system.

18 . The computer system of claim 17 wherein when the processor identifies, the program further causes the processor to at least one selected from the group consisting of: identify a border of predetermined color shown on the display device; identify four indicia of corner locations shown on the display device; and

identify a border of predetermined color that at least partially circumscribes the display device.

19 . A non-transitory computer readable medium storing a program that, when executed by a processor, causes the processor to:

receive an indication of gaze location of a user, the gaze location within a scene, and the scene including a display device associated with the processor;

receive an indication of resolution of the scene;

receive an indication of location of the display device within the scene;

perform a perspective transformation resulting in a desired location within the display device; and

move a cursor shown on the computer monitor to the desired location.

20 . The non-transitory computer readable medium of claim 19 wherein when the processor receives the indication of gaze location, the program causes the processor to receive a pixel location within the scene.

21 . The non-transitory computer readable medium of claim 19 wherein when processor receives the indication of resolution of the scene, the program causes the processor to receive an indication of pixel width and pixel height of the scene.

22 . The non-transitory computer readable medium of claim 19 wherein when processor receives the indication of location of the display device within the scene, the program causes the processor to receive four pixel locations within the scene corresponding to four corners of the display device within the scene.

23 . The non-transitory computer readable medium of claim 19 wherein the program further causes the processor to:

receive an indication from a headset system to begin a calibration procedure; and responsive to the indication from the headset display a plurality of calibration features on the display device.

24 . The non-transitory computer readable medium of claim 23 wherein when the processor displays, the program causes the processor to sequentially display nine calibration features at predetermined locations on the display device.