Granted Patent B2

US 12,353,617 · App. 17/620,004 · Granted Jul 8, 2025

Inventor: Moritz Kassner (Berlin, DE)

Assignee: Pupil Labs GmbH

G06F3/013G02B27/017G06V10/82G06V40/19G02B2027/0178

Abstract

Claims (68)

1. A system for determining one or more parameters of a user's eye, the system comprising:

a head-wearable device comprising a first camera comprising a field of view, and a non-transparent component, the first camera and the non-transparent component being arranged relative to each other so that at least a portion of the eye and at least a portion of the non-transparent component are in the field of view of the first camera when the user is wearing the head-wearable device; and

a computing unit implementing a neural network and configured to:

receive an image from the first camera, the image including at least the portion of the eye and the portion of the non-transparent component;

feed the image as an input into the neural network; and

determine the one or more parameters as an output of the neural network.

2. The system of claim 1 , wherein the head-wearable device comprises a second camera which is arranged with respect to the non-transparent component so that at least a portion of a further eye of the user and at least a further portion of the non-transparent component are in a field of view of the second camera when the user is wearing the head-wearable device, and wherein the computing unit is configured to:

receive an image from the second camera;

feed the image as an input into the neural network or another neural network; and

determine one or more parameters of the further eye as an output of the neural network.

3. The system of claim 2 , wherein a pose of at least one of the first camera and the second camera is adjustable, fixable and/or fixed with respect to the non-transparent component.

4. The system of claim 1 , wherein the non-transparent component is a main body of the head-wearable device, a nose bridge portion of the head-wearable device or a frame of the head-wearable device.

5. The system of claim 2 , wherein the each of the first and second cameras is arranged on the inner side of a frame of the head-wearable device, directly facing the user's eye such that a shortest optically possible path between the outer border of the eye and the outer border of the camera forms a straight line only intersecting air when the user is wearing the head-wearable device, wherein the respective camera is arranged in the nose bridge portion of the head-wearable device, and/or orientated within a range of 32 to 40, preferably, 34 to 38, in particular 36 degrees with respect to the median plane of the head-wearable device, and/or wherein the respective camera is arranged in a lateral portion of the head-wearable device, and/or located within a range of 108 to 125 degrees with respect to the median plane of the head-wearable device.

6. The system of claim 1 , wherein the one or more parameters comprise a gaze-direction related parameter, a gaze-related parameter and/or selected from a list consisting of: a gaze direction, a cyclopean gaze direction, a 3D gaze point, a 2D gaze point, a visual axis orientation, an optical axis orientation, a pupil axis orientation, a line of sight orientation, an orientation and/or a position and/or an eyelid closure, a pupil area, a pupil size, a pupil diameter, a sclera characteristic, an iris diameter, a characteristic of a blood vessel, a cornea characteristic of at least one eye, a cornea radius, an eyeball radius, a distance pupil-center to cornea-center, a distance cornea-center to eyeball-center, a distance pupil-center to limbus center, a cornea keratometric index of refraction, a cornea index of refraction, a vitreous humor index of refraction, a distance crystalline lens to eyeball-center, to cornea center and/or to corneal apex, a crystalline lens index of refraction, a degree of astigmatism, an orientation angle of a flat and/or a steep axis, a limbus major and/or minor axes orientation, an eye cyclo-torsion, an eye intra-ocular distance, an eye vergence, a statistics over eye adduction and/or eye abduction, and a statistics over eye elevation and/or eye depression, data about blink events, drowsiness and/or awareness of the user, parameters for user iris verification and/or identification.

7. The system of claim 1 , wherein the image is not pre-processed to obtain spatial and/or temporal patterns or arrangements prior to feeding it to the neural network, and/or wherein the image does not undergo feature extraction prior to feeding it to the neural network.

8. The system of claim 1 , wherein the output of the neural network is not post-processed for determining the one or more parameters, wherein the neural network comprises a two-dimensional input layer, wherein the respective image has a resolution of N×N pixels, and/or wherein the input layer is a N×N matrix or a N×2N matrix, wherein N is smaller or equal than 384, wherein the neural network comprises input nodes and/or an input layer for receiving values referring to at least one of a type of the camera, camera intrinsics of the camera, camera extrinsics of the camera, a type of the head-wearable device, and a size measure of or a deformation information about the head-wearable device, and/or wherein the neural network comprises at least one convolutional layer.

9. The system of claim 1 , wherein the computing unit is at least partly integrated into the head-wearable device and/or at least partly provided by a companion device of the system, in particular a mobile companion device.

10. The system of claim 1 , wherein the computing unit comprises:

one or more processors, in particular one or more CPUs, GPUs and/or DSPs; and

a neural network module comprising instructions which, when executed by at least one of the one or more processors, implement an instance of the neural network.

11. A method for determining one or more parameters of a user's eye, the method comprising:

providing a head-wearable device comprising a first camera comprising a field of view, and a non-transparent component, the first camera and the non-transparent component being arranged relative to each other so that at least a portion of the eye and at least a portion of the non-transparent component are in the field of view of the first camera when the user is wearing the head-wearable device;

creating an image using the first camera of the head-wearable device worn by the user;

feeding the image as an input into a neural network, the image including at least the portion of the eye and the portion of the non-transparent component; and

determining the one or more parameters as an output of the neural network.

12. The method of claim 11 , wherein the head-wearable device comprises a second camera which is arranged with respect to the non-transparent component so that at least a portion of a further eye of the user and at least a further portion of the non-transparent component are in a field of view of the second camera when the user is wearing the head-wearable device, further comprising:

creating an image using the second camera of the head-wearable device worn by the user; and

feeding the image as an input into the neural network.

13. The method of claim 11 , wherein the image is not pre-processed to obtain spatial and/or temporal patterns or arrangements prior to feeding it to the neural network, wherein the respective image does not undergo feature extraction prior to feeding it to the neural network, and/or wherein the output of the neural network is not post-processed for determining the one or more parameters.

14. The method of claim 11 , wherein, in addition to the image, at most data referring to a type of the camera, camera intrinsics of the camera, camera extrinsics of the camera, a type of the head-wearable device, and a size measure of or a deformation information about the head-wearable device is fed as an input into the neural network.

15. A method for creating and updating a database for training a neural network, in particular a convolutional neural, the method comprising:

presenting a first stimulus to a first user wearing a head-wearable device, the head-wearable device comprising a non-transparent component and a first camera arranged next to a first eye of the first user;

using the first camera of the head-wearable device to generate, when the first user responds to the first stimulus, a first image of at least a portion of the first eye of the first user and at least a portion of the non-transparent component;

establishing a data connection between the head-wearable device and the database;

generating a first dataset comprising the first image and a first representation of a gaze-related parameter, the first representation being correlated with the first stimulus; and

adding the first dataset to the database.

16. The method of claim 15 , the head-wearable device including a second camera arranged next to a second eye of the first user when the first user is wearing the head-wearable device, the method further comprising:

using, when the first user responds to the first stimulus, the second camera of the head-wearable device to generate a second image of at least a portion of the second eye of the first user and another portion of the non-transparent component, the first dataset further comprising the second image;

presenting a second stimulus to the first user wearing the head-wearable device;

using the first camera of the head-wearable device to generate, when the first user responds to the second stimulus, a third image of at least a portion of the first eye of the first user and at least the portion of the non-transparent component, and when the first user responds to the second stimulus, using the second camera of the head-wearable device to generate a fourth image of at least a portion of the second eye of the first user and the other portion of the non-transparent component;

establishing a data connection between the head-wearable device and the database;

generating a second dataset comprising the third image and the fourth image, and a second representation of the gaze-related parameter, the second representation being correlated with the second stimulus; and

adding the second dataset to the database.

17. The method of claim 15 , further comprising at least one of:

all of the following four steps:

presenting a third stimulus to a second user wearing the head-wearable device;

using the first camera of the head-wearable device to generate, when the second user responds to the third stimulus, a fifth image of at least a portion of a first eye of the second user and at least a portion the a non-transparent component, and when the second user responds to the third stimulus, using the second camera of the head-wearable device to generate a sixth image of at least a portion of a second eye of the second user and another portion of the non-transparent component;

generating a third dataset comprising the fifth image and the sixth image, and a third representation of the gaze-related parameter, the third representation being correlated with the third stimulus; and

adding the third dataset to the database, and

all of the following five steps:

presenting a fourth stimulus to the first user or the second user wearing a further head-wearable device comprising a non-transparent component and a first camera arranged next to a first eye of the user and a second camera arranged next to a second eye of the user when wearing the further head-wearable device;

using the first camera of the further head-wearable device to generate, when the user responds to the fourth stimulus, a seventh image of at least a portion of the first eye of the respective user and at least a portion the non-transparent component of the further head-wearable device, and when the respective user responds to the fourth stimulus, using the second camera of the further head-wearable device to generate an eighth image of at least a portion of a second eye of the respective user and another portion of the non-transparent component of the further head-wearable device;

establishing a data connection between the further head-wearable device and the database;

generating a fourth dataset comprising the seventh image, the eighth image, and a fourth representation of the gaze-related parameter, the fourth representation being correlated with the fourth stimulus; and

adding the fourth dataset to the database.

18. The method of claim 15 , further comprising:

using the first camera of the head-wearable device to generate a further image of at least a portion of the first eye of the user and at least the respective portion of the respective non-transparent component, and using the second camera of the head-wearable device to generate yet a further image of at least a portion of the second eye of the user and at least the portion of the respective non-transparent component;

generating a further dataset comprising the further image and the yet further image; and

adding the further dataset to the database.

19. The method of claim 15 , wherein establishing the data connection comprises connecting the head-wearable device with at least one of a desktop computer, a tablet, a laptop, a server, and a smartphone.

20. The method of claim 15 , wherein the stimulus comprises a visual stimulus, and/or wherein the stimulus comprises an acoustical stimulus, wherein the user is caused by acoustic stimulus to gaze at an object defining a gaze direction relative to a co-ordinate system fixed with the head-wearable device and/or a gaze point in the co-ordinate system, and/or wherein an image or a pair of images are generated when the user gazes at the object into the gaze direction and/or at the gaze point.

21. A system for determining one or more parameters of a user's eye, the system comprising:

a head-wearable device including a non-transparent component and at least a first camera having a field of view, the first camera and the non-transparent component arranged relative to each other so that at least a portion of the eye and at least a portion of the non-transparent component are in the field of view of the first camera when the user is wearing the head-wearable device, wherein the head-wearable device is implemented as a one of a spectacles device, an augmented reality device and a virtual reality device; and

a computing unit implementing a neural network and configured to:

receive an image from the first camera, the image including at least the portion of the eye and the portion of the non-transparent component;

feed the image as an input into the neural network; and

determine the one or more parameters as an output of the neural network.

22. The system of claim 21 , wherein the neural network is trained to take into account the portion of the eye with respect to the portion of the non-transparent component in the image to determine the one or more parameters of the user's eye.

Assignments (1)

Continuity (1)

References Cited (301)