Gaze estimation using one or more neural networks
Apparatuses, systems, and techniques are presented to estimate user gaze. In at least one embodiment, one or more neural networks are used to determine coarse and fine gaze estimates for one or more users.
1. One or more processors, comprising:
circuitry to use an amount by which a first image has changed from a second image to select whether to use one or more first portions of one or more neural networks to generate a coarse estimate of a location of one or more pupils within the first image, wherein one or more second portions of the one or neural networks are used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image when the amount is below a threshold amount.
2. The one or more processors of claim 1 , wherein the circuitry to use the one or more first portions of one or more neural networks to generate a coarse estimate of a location of one or more pupils further comprises passing a full eye image from one or more images of one or more eyes of one or more users through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
3. The one or more processors of claim 1 , wherein the circuitry to use the one or more second portions of one or more neural networks used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image further comprises passing smaller sub-regions of a full eye image through the one or more second portions of one or more neural networks, wherein the full eye image is the prior image that has passed through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
4. The one or more processors of claim 1 , wherein the one or more first portions of one or more neural networks and the one or more second portions of one or more neural networks use a same convolutional neural network (CNN)-based architecture but with different network parameters.
5. The one or more processors of claim 1 , wherein the threshold amount comprises an amount of lateral movement of a pupil in an image.
6. The one or more processors of claim 1 , wherein the second image includes one or more separate images for both eyes of one of one or more users, and wherein the circuitry is further to concatenate those separate images into a concatenated image with padding between representations of the eyes before providing the concatenated image as input to the one or more first portions of the one or more neural networks.
7. A system comprising:
one or more processors to use an amount by which a first image has changed from a second image to select whether to use one or more first portions of one or more neural networks to generate a coarse estimate of a location of one or more pupils within the first image, wherein one or more second portions of the one or neural networks are used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image when the amount is below a threshold amount.
8. The system of claim 7 , wherein the one or more processors to use one or more first portions of the one or more neural networks to generate a coarse estimate of a location of one or more pupils further comprises passing a full eye image from one or more images of one or more eyes of one or more users through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
9. The system of claim 7 , wherein the one or more processors to use one or more second portions of the one or more neural networks used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image further comprises passing smaller sub-regions of a full eye image through the one or more second portions of one or more neural networks, wherein the full eye image is the prior image that has passed through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
10. The system of claim 7 , wherein the one or more first portions of one or more neural networks and the one or more second portions of one or more neural networks use a same convolutional neural network (CNN)-based architecture but with different network parameters.
11. The system of claim 7 , wherein the threshold amount comprises an amount of lateral movement of a pupil in an image.
12. The system of claim 7 , wherein the second image includes one or more separate images for both eyes of one of one or more users, and wherein the one or more processors are further to concatenate those separate images into a concatenated image with padding between representations of the eyes before providing the concatenated image as input to the one or more first portions of the one or more neural networks.
13. A method comprising:
using an amount by which a first image has changed from a second image to select whether to use one or more first portions of one or more neural networks to generate a coarse estimate of a location of one or more pupils within the first image, wherein one or more second portions of the one or neural networks are used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image when the amount is below a threshold amount.
14. The method of claim 13 , wherein using the one or more first portions of one or more neural networks to generate a coarse estimate of a location of one or more pupils further comprises passing a full eye image from one or more images of one or more eyes of one or more users through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
15. The method of claim 13 , wherein using the one or more second portions of one or more neural networks to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image further comprises passing smaller sub-regions of a full eye image through the one or more second portions of one or more neural networks, wherein the full eye image is the prior image that has passed through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
16. The method of claim 13 , wherein the one or more first portions of one or more neural networks and the one or more second portions of one or more neural networks use a same convolutional neural network (CNN)-based architecture but with different network parameters.
17. The method of claim 13 , wherein
threshold amount comprises an amount of lateral movement of a pupil in an image.
18. The method of claim 13 , wherein the second image includes one or more separate images for both eyes of one of one or more users, and further comprising concatenating those separate images into a concatenated image with padding between representations of the eyes before providing the concatenated image as input to the one or more first portions of the one or more neural networks.
19. A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
use an amount by which a first image has changed from a second image to select whether to use one or more first portions of one or more neural networks to generate a coarse estimate of a location of one or more pupils within the first image, wherein one or more second portions of the one or neural networks are used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image when the amount is below a threshold amount.
20. The non-transitory machine-readable medium of claim 19 , wherein the one or more processors to use one or more first portions of the one or more neural networks to generate a coarse estimate of a location of one or more pupils further comprises passing a full eye image from one or more images of one or more eyes of one or more users through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
21. The non-transitory machine-readable medium of claim 19 , wherein the one or more processors to use one or more second portions of the one or more neural networks used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image further comprises passing smaller sub-regions of a full eye image through the one or more second portions of one or more neural networks, wherein the full eye image is the prior image that has passed through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
22. The non-transitory machine-readable medium of claim 19 , wherein the one or more first portions of one or more neural networks and the one or more second portions of one or more neural networks use a same convolutional neural network (CNN)-based architecture but with different network parameters.
23. The non-transitory machine-readable medium of claim 19 , wherein the
threshold amount comprises an amount of lateral movement of a pupil in an image.
24. The non-transitory machine-readable medium of claim 20 , wherein the second image includes one or more separate images for both eyes of one of one or more users, and wherein the one or more processors are further to concatenate those separate images into a concatenated image with padding between representations of the eyes before providing the concatenated image as input to the one or more first portions of the one or more neural networks.
25. A gaze estimation system, comprising:
one or more processors to use an amount by which a first image has changed from a second image to select whether to use one or more first portions of one or more neural networks to generate a coarse estimate of a location of one or more pupils within the first image, wherein one or more second portions of the one or neural networks are used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image when the amount is below a threshold amount.
26. The gaze estimation system of claim 25 , wherein the one or more processors to use one or more first portions of the one or more neural networks to generate a coarse estimate of a location of one or more pupils further comprises passing a full eye image from one or more images of one or more eyes of one or more users through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
27. The gaze estimation system of claim 25 , wherein the one or more processors to use one or more second portions of the one or more neural networks used to generate a fine estimate of the location of the one or more pupils within the first image using a prior estimate from the second image further comprises passing smaller sub-regions of a full eye image through the one or more second portions of one or more neural networks, wherein the full eye image is the prior image that has passed through the one or more first portions of one or more neural networks to generate smaller sub-regions of the full eye image.
28. The gaze estimation system of claim 25 , wherein the one or more first portions of one or more neural networks and the one or more second portions of one or more neural networks use a same convolutional neural network (CNN)-based architecture but with different network parameters.
29. The gaze estimation system of claim 25 , wherein the threshold amount comprises an amount of lateral movement of a pupil in an image.
30. The gaze estimation system of claim 25 , wherein the second image includes one or more separate images for both eyes of one of one or more users, and wherein the one or more processors are further to concatenate those separate images into a concatenated image with padding between representations of the eyes before providing the concatenated image as input to the one or more first portions of the one or more neural networks.
31. The one or more processors of claim 1 , wherein the circuitry to select whether to use the one or more first portions of the one or more neural networks, is further to use only the one or more second portions of the one or more neural networks based on the threshold amount, wherein the threshold amount comprises the amount by which the pupil location changes from the first image to the second image.