IP Library › Granted Patent US 11,450,113

Granted Patent B1

US 11,450,113 · App. 16/578,285 · Granted Sep 20, 2022

Method and apparatus for a wearable computer

Inventor: Masoud Vaziri (Richardson, TX)

G06V20/52 G02B13/16 G02B26/0875 G02B27/0093 G02B27/017 G02B27/0172 G02B27/64 G06F1/163 G06F3/012 G06F3/013 G06F3/015 G06F3/017 G06F3/0484 G06F3/04842 G06F3/167 G06V20/597 G06V40/19 H04N5/2256 H04N5/2259 H04N5/332 H04N7/185 H04R1/028 H04W4/023 G02B2027/014 G02B2027/0132 G02B2027/0138 G02B2027/0178 G02B2027/0187 G06F2203/04806 H04R2201/107 H04R2460/13

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,450,113
App. No.: 16/578,285
Granted: Sep 20, 2022
Kind: B1

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

An embodiment of a Wearable Computer apparatus includes a first portable unit for data gathering and providing a natural user interface, and a second portable unit for processing the gathered data from the first unit and taking an action in respond to the received data. The first portable unit includes an eyeglass frame, at least one first scene camera disposed on the eyeglass frame for capturing at least one scene image corresponding to a field of view of a user, at least one microphone, one speaker and one LED to create a natural user interface, and at least one first processor to receive data from the data gathering units in the first portable unit and communicating that data to the second portable unit. The second portable unit is in communication with the first portable unit and includes at least one second processor configured for receiving data from the first processor. The second portable unit also includes at least one interface of a digital personal assistant that receives at least one scene image from the first portable unit and initiates an object recognition procedure to recognize an object in the at least one scene image. Based on the at least one recognized object, the digital personal assistant takes an action that may include providing a feedback to the user via light or audio.

Claims (236)

1. A multi-camera imaging apparatus comprising:

optical and sensing elements configured to be worn by a user, the optical and sensing elements comprise:

a frame configured to be worn on the head of the user;

at least one scene camera module disposed on or in the frame and configured to capture scene images of a scene viewed by the user;

at least one eye tracking module disposed on or in the frame and configured to capture eye images of the eyes of the user; and

at least one motion sensor disposed on or in the frame to capture head movements data representing head movements of the user;

electronics elements configured to be carried or worn by the user, the electronics elements comprise at least one processor configured to:

receive the scene images from the at least one scene camera module;

recognize at least one object in the scene images to provide at least one object recognition;

receive the eye images from the at least one eye tracking module;

estimate eye movements of the eyes of the user based on the eye images;

maintain an eye movements history of the eye movements over time;

receive the head movements data from the at least one motion sensor;

maintain a head movements history of the head movements over time based on the head movements data; and

perform an operation based on at least a portion of the head movements history, at least a portion of the eye movements history, and the at least one object recognition.

2. The multi-camera imaging apparatus of claim 1 wherein the perform the operation comprises determining that the user is paying attention to or looking at the at least one object using the at least one object recognition, the at least a portion of the head movements history, and the at least a portion of the eye movements history.

3. The multi-camera imaging apparatus of claim 2 wherein:

the at least one processor is further configured to generate a visual feedback signal representing a visual feedback indication that the user is paying attention to or looking at the at least one object;

the optical and sensing elements further comprise a feedback unit disposed on or in the frame and configured to:

receive the visual feedback signal from the at least one processor; and

display the visual feedback indication to the eyes of the user in response to the visual feedback signal.

4. The multi-camera imaging apparatus of claim 3 wherein:

the visual feedback indication comprises a specific color of light; and

the feedback unit comprises light emitting diodes that emit the specific color of light.

5. The multi-camera imaging apparatus of claim 2 wherein the determining that the user is paying attention to or looking at the at least one object comprises determining that the eye movements of the user maintained in the at least portion of the eye movements history have only a small variation or spread while the head movements of the user maintained in the at least portion of the head movements history remain relatively fixed.

6. The multi-camera imaging apparatus of claim 5 wherein the eye movements of the user comprise gaze directions of the user.

7. The multi-camera imaging apparatus of claim 2 wherein the determining that the user is paying attention to or looking at the at least one object comprises generating a motion vector representing the gaze directions of the user by combining the at least portion of the eye movements history and the at least portion of the head movements history.

8. The multi-camera imaging apparatus of claim 7 wherein:

the head movements maintained in the head movements history comprises linear head accelerations and angular head rotations;

the eye movements maintained in the eye movements history comprises angular eye rotations;

the generating the motion vector comprises:

generating a linear head acceleration vector for the linear head accelerations in the at least portion of the head movements history;

generating an angular head rotation vector for the angular head rotations in the at least portion of the head movements history;

generating an angular eye rotation vector for the angular eye rotations in the at least portion of the eye movements history; and

summing the linear head acceleration vector, the angular head rotation vector, and the angular eye rotation vector to form the motion vector.

9. The multi-camera imaging apparatus of claim 2 wherein:

the at least one object comprises a hand of the user;

the determining that the user is paying attention to or looking at the at least one object comprises determining that the user is looking at the hand of the user;

and the operation further comprises interpreting a gesture or a movement of the hand of the user in at least some of the scene images.

10. The multi-camera imaging apparatus of claim 9 wherein:

the at least one processor is further configured to generate a visual feedback signal representing a visual feedback indication that the user is looking at the hand;

the optical and sensing elements further comprise a feedback unit disposed on or in the frame and configured to:

receive the visual feedback signal from the at least one processor; and

display the visual feedback indication to the eyes of the user in response to the visual feedback signal.

11. The multi-camera imaging apparatus of claim 9 wherein:

the at least one processor is further configured to generate a visual feedback signal representing a visual feedback indication that the gesture or the movement of the hand is understood;

the optical and sensing elements further comprise a feedback unit disposed on or in the frame and configured to:

receive the visual feedback signal from the at least one processor; and

display the visual feedback indication to the eyes of the user in response to the visual feedback signal.

12. The multi-camera imaging apparatus of claim 9 wherein the perform the operation further comprises selecting subset portions of at least some of the scene images containing the hand of the user and interpreting the gesture or the movement of the hand of the user in the subset portions of the at least some of the scene images.

13. The multi-camera imaging apparatus of claim 1 wherein the perform the operation includes anticipating a need of the user with respect to the at least one object using at least a portion of the head movements history and at least a portion of the eye movements history.

14. The multi-camera imaging apparatus of claim 1 wherein the eye movements history provides a trajectory of the eye movements of the eyes of the user.

15. The multi-camera imaging apparatus of claim 2 wherein the perform the operation further comprises creating a scene map of the at least one object based on the determining the user is paying attention to or looking at the at least one object.

16. The multi-camera imaging apparatus of claim 1 wherein the at least one processor comprises:

at least one first processor configured to:

receive the scene images from the at least one scene camera module;

recognize the at least one object in the scene images to provide the at least one object recognition;

receive the eye images from the at least one eye tracking module;

estimate the eye movements of the eyes of the user based on the eye images;

maintain the eye movements history of the eye movements over time;

receive the head movements data from the at least one motion sensor; and

maintain the head movements history of the head movements over time based on the head movements data; and

at least one second processor configured to:

receive the head movements history, the eye movements history, and the at least one object recognition from the at least one first processor;

perform the operation based on the at least a portion of the head movements history, the at least a portion of the eye movements history, and the at least one object recognition.

17. The multi-camera imaging apparatus of claim 1 wherein first ones of the electronics elements are configured to be worn by the user and are disposed on or in the frame and second ones of the electronics elements are configured to be carried by the user and are disposed in a smartphone.

18. The multi-camera imaging apparatus of claim 17 wherein:

the first ones of the electronics elements comprise at least one first processor of the at least one processors that is configured to:

receive the scene images from the at least one scene camera module;

recognize the at least one object in the scene images to provide the at least one object recognition;

receive the eye images from the at least one eye tracking module;

estimate the eye movements of the eyes of the user based on the eye images;

maintain the eye movements history of the eye movements over time;

receive the head movements data from the at least one motion sensor; and

maintain the head movements history of the head movements over time based on the head movements data; and

the second ones of the electronics elements comprise at least one second processor of the at least one processors that is configured to:

receive the head movements history, the eye movements history, and the at least one object from the at least one first processor;

perform the operation based on the at least a portion of the head movements history, the at least a portion of the eye movements history, and the at least one object recognition.

19. A multi-camera imaging apparatus comprising:

optical and sensing elements configured to be worn by a user, the optical and sensing elements comprise:

a frame configured to be worn on the head of the user;

at least one scene camera module disposed on or in the frame and configured to capture scene images of a scene viewed by the user;

at least one eye tracking module disposed on or in the frame and configured to capture eye images of the eyes of the user; and

at least one motion sensor disposed on or in the frame to capture head movements data representing head movements of the user;

electronics elements configured to be carried or worn by the user, the electronics elements comprise at least one first processor configured to:

receive the scene images from the at least one scene camera module;

recognize at least one object in the scene images to provide at least one object recognition;

receive the eye images from the at least one eye tracking module;

estimate eye movements of the eyes of the user based on the eye images;

maintain an eye movements history of the eye movements over time;

receive the head movements data from the at least one motion sensor;

maintain a head movements history of the head movements over time based on the head movements data; and

provide at least a portion of the head movements history, at least a portion of the eye movements history, and the at least one object recognition for use by software that performs an operation based thereon.

20. The multi-camera imaging apparatus of claim 19 wherein the perform the operation comprises determining that the user is paying attention to or looking at the at least one object using the at least one object recognition, the at least a portion of the head movements history, and the at least a portion of the eye movements history.

21. The multi-camera imaging apparatus of claim 20 wherein:

the optical and sensing elements further comprise a feedback unit disposed on or in the frame and configured to:

receive the visual feedback signal from the at least one processor; and

display the visual feedback indication to the eyes of the user in response to the visual feedback signal.

22. The multi-camera imaging apparatus of claim 21 wherein:

the visual feedback indication comprises a specific color of light; and

the feedback unit comprises light emitting diodes that emit the specific color of light.

23. The multi-camera imaging apparatus of claim 20 wherein the determining that the user is paying attention to or looking at the at least one object comprises determining that the eye movements of the user maintained in the at least portion of the eye movements history have only a small variation or spread while the head movements of the user maintained in the at least portion of the head movements history remain relatively fixed.

24. The multi-camera imaging apparatus of claim 23 wherein the eye movements of the user comprise gaze directions of the user.

25. The multi-camera imaging apparatus of claim 20 wherein the determining that the user is paying attention to or looking at the at least one object comprises generating a motion vector representing the gaze directions of the user by combining the at least portion of the eye movements history and the at least portion of the head movements history.

26. The multi-camera imaging apparatus of claim 25 wherein:

the head movements maintained in the head movements history comprises linear head accelerations and angular head rotations;

the eye movements maintained in the eye movements history comprises angular eye rotations;

the generating the motion vector comprises:

generating a linear head acceleration vector for the linear head accelerations in the at least portion of the head movements history;

generating an angular head rotation vector for the angular head rotations in the at least portion of the head movements history;

generating an angular eye rotation vector for the angular eye rotations in the at least portion of the eye movements history; and

summing the linear head acceleration vector, the angular head rotation vector, and the angular eye rotation vector to form the motion vector.

27. The multi-camera imaging apparatus of claim 20 wherein:

the at least one object comprises a hand of the user;

the determining that the user is paying attention to or looking at the at least one object comprises determining that the user is looking at the hand of the user;

and the operation further comprises interpreting a gesture or a movement of the hand of the user in at least some of the scene images.

28. The multi-camera imaging apparatus of claim 27 wherein:

the at least one processor is further configured to generate a visual feedback signal representing a visual feedback indication that the user is looking at the hand;

the optical and sensing elements further comprise a feedback unit disposed on or in the frame and configured to:

receive the visual feedback signal from the at least one processor; and

display the visual feedback indication to the eyes of the user in response to the visual feedback signal.

29. The multi-camera imaging apparatus of claim 27 wherein:

the at least one processor is further configured to generate a visual feedback signal representing a visual feedback indication that the gesture or the movement of the hand is understood;

the optical and sensing elements further comprise a feedback unit disposed on or in the frame and configured to:

receive the visual feedback signal from the at least one processor; and

display the visual feedback indication to the eyes of the user in response to the visual feedback signal.

30. The multi-camera imaging apparatus of claim 27 wherein the perform the operation further comprises selecting subset portions of at least some of the scene images containing the hand of the user and interpreting the gesture or the movement of the hand of the user in the subset portions of the at least some of the scene images.

31. The multi-camera imaging apparatus of claim 19 wherein the perform the operation includes anticipating a need of the user with respect to the at least one object using at least a portion of the head movements history and at least a portion of the eye movements history.

32. The multi-camera imaging apparatus of claim 19 wherein the eye movements history provides a trajectory of the eye movements of the eyes of the user.

33. The multi-camera imaging apparatus of claim 19 wherein the perform the operation further comprises creating a scene map of the at least one object based on the determining the user is paying attention to or looking at the at least one object.

34. The multi-camera imaging apparatus of claim 19 wherein the at least one processor comprises:

at least one first processor configured to:

receive the scene images from the at least one scene camera module;

recognize the at least one object in the scene images to provide the at least one object recognition;

receive the eye images from the at least one eye tracking module;

estimate the eye movements of the eyes of the user based on the eye images;

maintain the eye movements history of the eye movements over time;

receive the head movements data from the at least one motion sensor; and

maintain the head movements history of the head movements over time based on the head movements data; and

at least one second processor configured to:

receive the head movements history, the eye movements history, and the at least one object recognition from the at least one first processor;

provide the at least a portion of the head movements history, the at least a portion of the eye movements history, and the at least one object recognition for use by the software to perform the operation based thereon.

35. The multi-camera imaging apparatus of claim 19 wherein first ones of the electronics elements are configured to be worn by the user and are disposed on or in the frame and second ones of the electronics elements are configured to be carried by the user and are disposed in a smartphone.

36. The multi-camera imaging apparatus of claim 35 wherein: