Method and apparatus for a wearable computer
An embodiment of a Wearable Computer apparatus includes a first portable unit for data gathering and providing a natural user interface, and a second portable unit for processing the gathered data from the first unit and taking an action in respond to the received data. The first portable unit includes an eyeglass frame, at least one first scene camera disposed on the eyeglass frame for capturing at least one scene image corresponding to a field of view of a user, at least one microphone, one speaker and one LED to create a natural user interface, and at least one first processor to receive data from the data gathering units in the first portable unit and communicating that data to the second portable unit. The second portable unit is in communication with the first portable unit and includes at least one second processor configured for receiving data from the first processor. The second portable unit also includes at least one interface of a digital personal assistant that receives at least one scene image from the first portable unit and initiates an object recognition procedure to recognize an object in the at least one scene image. Based on the at least one recognized object, the digital personal assistant takes an action that may include providing a feedback to the user via light or audio.
1. An imaging apparatus comprising:
a frame configured to be worn on the head of a user;
a scene camera module coupled to the frame and configured to capture scene images of a scene viewed by the user;
an eye tracking module disposed on or in the frame and configured to capture eye images of the eyes of the user; and
a motion sensor disposed on or in the frame to capture head movements data representing head movements of the user;
a processor configured to:
receive the scene images from the scene camera module;
recognize an object in the scene images to provide object recognition;
receive the eye images from the eye tracking module;
determine the user is authorized to use the imaging apparatus based at least in part on recognizing the eyes of the user in first ones of the eye images;
estimate eye movements of the eyes of the user based on second ones of the eye images;
maintain an eye movements history of the eye movements over time;
receive the head movements data from the motion sensor;
maintain a head movements history of the head movements over time based on the head movements data; and
perform an operation based on at least a portion of the head movements history, at least a portion of the eye movements history, and the object recognition.
2. The imaging apparatus of claim 1 wherein the perform the operation comprises determining that the user is paying attention to or looking at the object using the object recognition, the at least a portion of the head movements history, and the at least a portion of the eye movements history.
3. The imaging apparatus of claim 2 wherein:
the processor is further configured to generate a visual feedback signal representing a visual feedback indication that the user is paying attention to or looking at the object;
the imaging apparatus further comprises a feedback unit disposed on or in the frame and configured to:
receive the visual feedback signal from the processor; and
display the visual feedback indication to the eyes of the user in response to the visual feedback signal.
4. The imaging apparatus of claim 3 wherein:
the visual feedback indication comprises a specific color of light; and
the feedback unit comprises light emitting diodes that emit the specific color of light.
5. The imaging apparatus of claim 2 wherein the determining that the user is paying attention to or looking at the object comprises determining that the eye movements of the user maintained in the at least a portion of the eye movements history have only a small variation or spread while the head movements of the user maintained in the at least portion of the head movements history remain relatively fixed.
6. The imaging apparatus of claim 5 wherein the eye movements of the user comprise gaze directions of the user.
7. The imaging apparatus of claim 2 wherein the determining that the user is paying attention to or looking at the object comprises generating a motion vector representing the gaze directions of the user by combining the at least portion of the eye movements history and the at least portion of the head movements history.
8. The imaging apparatus of claim 7 wherein:
the head movements maintained in the head movements history comprise linear head accelerations and angular head rotations;
the eye movements maintained in the eye movements history comprise angular eye rotations;
the generating the motion vector comprises:
generating a linear head acceleration vector for the linear head accelerations in the at least portion of the head movements history;
generating an angular head rotation vector for the angular head rotations in the at least portion of the head movements history;
generating an angular eye rotation vector for the angular eye rotations in the at least portion of the eye movements history; and
summing the linear head acceleration vector, the angular head rotation vector, and the angular eye rotation vector to form the motion vector.
9. The imaging apparatus of claim 2 wherein:
the object comprises a hand of the user;
the determining that the user is paying attention to or looking at the object comprises determining that the user is looking at the hand of the user;
and the operation further comprises interpreting a gesture or a movement of the hand of the user in at least some of the scene images.
10. The imaging apparatus of claim 9 wherein:
the processor is further configured to generate a visual feedback signal representing a visual feedback indication that the user is looking at the hand;
the imaging apparatus further comprises a feedback unit disposed on or in the frame and configured to:
receive the visual feedback signal from the processor; and
display the visual feedback indication to the eyes of the user in response to the visual feedback signal.
11. The imaging apparatus of claim 9 wherein:
the processor is further configured to generate a visual feedback signal representing a visual feedback indication that the gesture or the movement of the hand is understood;
the imaging apparatus further comprises a feedback unit disposed on or in the frame and configured to:
receive the visual feedback signal from the processor; and
display the visual feedback indication to the eyes of the user in response to the visual feedback signal.
12. The imaging apparatus of claim 9 wherein the perform the operation further comprises selecting subset portions of at least some of the scene images containing the hand of the user and interpreting the gesture or the movement of the hand of the user in the subset portions of the at least some of the scene images.
13. The imaging apparatus of claim 1 wherein the perform the operation includes anticipating a need of the user with respect to the object using at least a portion of the head movements history and at least a portion of the eye movements history.
14. The imaging apparatus of claim 1 wherein the eye movements history provides a trajectory of the eye movements of the eyes of the user.
15. The imaging apparatus of claim 2 wherein the perform the operation further comprises creating a scene map of the object based on the determining the user is paying attention to or looking at the object.
16. The imaging apparatus of claim 1 wherein the processor comprises:
a first processor configured to:
receive the scene images from the scene camera module;
recognize the object in the scene images to provide the object recognition;
receive the eye images from the eye tracking module;
determine the user is authorized to use the imaging apparatus based at least in part on recognizing the eyes of the user in first ones of the eye images;
estimate eye movements of the eyes of the user based on second ones of the eye images;
maintain the eye movements history of the eye movements over time;
receive the head movements data from the motion sensor; and
maintain the head movements history of the head movements over time based on the head movements data; and
a second processor configured to:
receive the head movements history, the eye movements history, and the object recognition from the first processor;
perform the operation based on the at least a portion of the head movements history, the at least a portion of the eye movements history, and the object recognition.
17. The imaging apparatus of claim 1 wherein
the operation is performed by providing the at least a portion of the head movements history, the at least a portion of the eye movements history, and the object recognition for use by software that performs the operation based thereon.
18. The imaging apparatus of claim 17 wherein the imaging apparatus further comprises the software.
19. The imaging apparatus of claim 1 wherein
the operation is performed by executing software that performs the operation based on the at least a portion of the head movements history, the at least a portion of the eye movements history, and the object recognition.
20. The imaging apparatus of claim 19 wherein the imaging apparatus further comprises the software.
21. The imaging apparatus of claim 1 , wherein the scene camera module is disposed on or in the frame.
22. The imaging apparatus of claim 1 , wherein the scene camera module is remotely coupled to the frame.
23. The imaging apparatus of claim 1 , further comprising electronics elements configured to be disposed on or in the frame, wherein the electronics elements comprise the processor.
24. The imaging apparatus of claim 1 , further comprising electronics elements configured to be remotely coupled to the frame, wherein the electronics elements comprise the processor.
25. A method for performing an operation on an object, the method comprising:
providing an imaging apparatus, comprising:
a frame configured to be worn on the head of a user;
a scene camera module coupled to the frame and configured to capture scene images of a scene viewed by the user;
an eye tracking module disposed on or in the frame and configured to capture eye images of the eyes of the user;
a motion sensor disposed on or in the frame to capture head movements data representing head movements of the user; and
a processor;
receiving the scene images from the scene camera module;
recognizing the object in the scene images to provide object recognition;
receiving the eye images from the eye tracking module;
determining the user is authorized to use the imaging apparatus based at least in part on recognizing the eyes of the user in first ones of the eye images;
estimating eye movements of the eyes of the user based on second ones of the eye images;
maintaining an eye movements history of the eye movements over time;
receiving the head movements data from the motion sensor;
maintaining a head movements history of the head movements over time based on the head movements data; and
performing the operation based on at least a portion of the head movements history, at least a portion of the eye movements history, and the object recognition.
26. The method of claim 25 wherein the performing the operation comprises determining that the user is paying attention to or looking at the object using the object recognition, the at least a portion of the head movements history, and the at least a portion of the eye movements history.
27. The method of claim 26 wherein the determining that the user is paying attention to or looking at the object comprises generating a motion vector representing the gaze directions of the user by combining the at least a portion of the eye movements history and the at least a portion of the head movements history.
28. The method of claim 27 , wherein:
the head movements maintained in the head movements history comprise linear head accelerations and angular head rotations;
the eye movements maintained in the eye movements history comprise angular eye rotations; and
the generating the motion vector comprises:
generating a linear head acceleration vector for the linear head accelerations in the at least a portion of the head movements history;
generating an angular head rotation vector for the angular head rotations in the at least a portion of the head movements history;
generating an angular eye rotation vector for the angular eye rotations in the at least a portion of the eye movements history; and
summing the linear head acceleration vector, the angular head rotation vector, and the angular eye rotation vector to form the motion vector.