IP Library › Granted Patent US 10,666,918
Granted Patent B2
US 10,666,918 · App. 16/538,057 · Granted May 26, 2020

Vision-based alerting based on physical contact prediction

Inventor: Mohamed Abou Shousha (Pembroke Pines, FL)
Assignee: University of Miami
H04N9/646A61B3/0025A61B3/032A61B3/113A61B3/14G02B27/0172G06K9/0061G06K9/00671G06K9/6256G06T5/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,666,918
App. No.
16/538,057
Granted
May 26, 2020
Kind
B2
Abstract

In certain embodiments, vision-related alerting may be facilitated based on a physical contact prediction. In some embodiments, a prediction indicating defective visual field portions of a user's visual field may be generated based on feedback related to a set of stimuli displayed to the user. The defective visual field portions may be visual field portions of the visual field that fail to satisfy one or more vision criteria. Based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be presented via a wearable device. Activation of an alert may be caused via the wearable device based on (i) a prediction of physical contact (of an object with the user) and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one visual field portion satisfying the vision criteria.

Claims (56)

1. A system for facilitating vision-related alerting based on a physical contact prediction, the system comprising:

a computer system that comprises one or more processors executing computer program instructions that, when executed, cause the computer system to:

obtain, via a wearable device, feedback related to a set of stimuli displayed to a user during a visual test presentation, the feedback related to the set of stimuli indicating whether or how the user sees one or more stimuli of the set, the feedback comprising: (i) an eye image captured during the visual test presentation, the eye image being an image of an eye of the user, (ii) an eye response of an eye of the user to one or more stimuli, or (iii) a user modification of stimuli;

generate a prediction indicating defective visual field portions of a visual field of the user based on the eye image, the eye response, or the user modification of stimuli, wherein the visual field of the user comprises visual field portions, the defective visual field portions being ones of the visual field portions that fail to satisfy one or more vision criteria;

cause, based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be displayed on one or more other display portions of the wearable device, the live video stream representing an environment of the user;

detect an object in the defective visual field portions or predicted to be in the defective visual field portions;

generating a prediction indicating that the object will come in physical contact with the user; and

causing an alert to be presented via the wearable device based on (i) the prediction of physical contact and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one of the visual field portions satisfying the one or more vision criteria.

2. The system of claim 1 , wherein the feedback related to the set of stimuli comprises the eye image captured during the visual test presentation, and the eye image is an image of a retina of the eye or an image of a cornea of the eye, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye image, the prediction indicating the defective visual field portions.

3. The system of claim 1 , wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises an eye movement, a gaze direction, or a pupil size change, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye response to one or more stimuli, the prediction indicating the defective visual field portions.

4. The system of claim 1 , wherein the feedback related to the set of stimuli comprises the user modification of one or more stimuli via user input of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user modification of one or more stimuli, the prediction indicating the defective visual field portions.

5. A method being implemented by one or more processors executing computer program instructions that, when executed, perform the method, the method comprising:

obtaining feedback related to a set of stimuli displayed to a user, the feedback related to the set of stimuli comprising: (i) an eye image captured during a visual test presentation, the eye image being an image of an eye of the user, (ii) an eye response of an eye of the user to one or more stimuli, or (iii) a user modification of stimuli;

generate a prediction indicating defective visual field portions of a visual field of the user based on the eye image, the eye response, or the user modification of stimuli, wherein the visual field of the user comprises visual field portions, the defective visual field portions being ones of the visual field portions that fail to satisfy one or more vision criteria;

cause, based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be displayed on one or more other display portions of a wearable device, the live video stream representing an environment of the user;

detect an object in the defective visual field portions or predicted to be in the defective visual field portions;

generating a prediction indicating that the object will come in physical contact with the user; and

causing an alert to be presented via the wearable device based on (i) the prediction of physical contact and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one of the visual field portions satisfying the one or more vision criteria.

6. The method of claim 5 , wherein the feedback related to the set of stimuli comprises the eye image captured during the visual test presentation, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye image, the prediction indicating the defective visual field portions.

7. The method of claim 6 , wherein the eye image is an image of a cornea of the eye, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the image of the cornea, the prediction indicating the defective visual field portions.

8. The method of claim 5 , wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises an eye movement of an eye of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye movement, the prediction indicating the defective visual field portions.

9. The method of claim 5 , wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises a gaze direction of an eye of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the gaze direction, the prediction indicating the defective visual field portions.

10. The method of claim 5 , wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises a pupil size change of an eye of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the pupil size change, the prediction indicating the defective visual field portions.

11. The method of claim 5 , wherein the feedback related to the set of stimuli comprises the user modification of one or more stimuli via user input of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user modification of one or more stimuli, the prediction indicating the defective visual field portions.

12. The method of claim 11 , wherein the user modification of one or more stimuli comprises a user manipulation of a graphical representation of one or more stimuli, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user manipulation of the graphical representation, the prediction indicating the defective visual field portions.

13. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause operations comprising:

obtaining feedback related to a set of stimuli displayed to a user, the feedback related to the set of stimuli indicating whether or how the user sees one or more stimuli of the set;

generate a prediction indicating defective visual field portions of a visual field of the user based on the feedback related to the set of stimuli, wherein the visual field of the user comprises visual field portions, the defective visual field portions being ones of the visual field portions that fail to satisfy one or more vision criteria;

cause, based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be presented via a wearable device, the live video stream representing an environment of the user;

detect an object in the defective visual field portions or predicted to be in the defective visual field portions;

generating a prediction indicating that the object will come in physical contact with the user; and

causing activation of an alert via the wearable device based on (i) the prediction of physical contact and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one of the visual field portions satisfying the one or more vision criteria.

14. The one or more non-transitory computer-readable media of claim 13 , wherein the feedback related to the set of stimuli comprises an eye image captured during a visual test presentation, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye image, the prediction indicating the defective visual field portions.

15. The one or more non-transitory computer-readable media of claim 14 , wherein the eye image is an image of a cornea of an eye of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the image of the cornea, the prediction indicating the defective visual field portions.

16. The one or more non-transitory computer-readable media of claim 13 , wherein the feedback related to the set of stimuli comprises an eye response of an eye of the user to one or more stimuli, and the eye response comprises an eye movement of an eye of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye movement, the prediction indicating the defective visual field portions.

17. The one or more non-transitory computer-readable media of claim 13 , wherein the feedback related to the set of stimuli comprises an eye response of an eye of the user to one or more stimuli, and the eye response comprises a gaze direction of an eye of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the gaze direction, the prediction indicating the defective visual field portions.

18. The one or more non-transitory computer-readable media of claim 13 , wherein the feedback related to the set of stimuli comprises an eye response of an eye of the user to one or more stimuli, and the eye response comprises a pupil size change of an eye of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the pupil size change, the prediction indicating the defective visual field portions.

19. The one or more non-transitory computer-readable media of claim 13 , wherein the feedback related to the set of stimuli comprises a user modification of one or more stimuli via user input of the user, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user modification of one or more stimuli, the prediction indicating the defective visual field portions.

20. The one or more non-transitory computer-readable media of claim 19 , wherein the user modification of one or more stimuli comprises a user manipulation of a graphical representation of one or more stimuli, and

wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user manipulation of the graphical representation, the prediction indicating the defective visual field portions.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 12, 2019
From: ABOU SHOUSHA, MOHAMED
To: UNIVERSITY OF MIAMI
Reel/Frame 050025/0666 →
Continuity (5)
Continuation 16428932 · May 31, 2019
Continuation 16367633 · Mar 28, 2019
Continuation In Part 16144995 · Sep 27, 2018
Provisional Application 62563770 · Sep 27, 2017
Related Publication 20190379869A1 · Dec 12, 2019
Cited By (1)
US 12,446,770