Method and apparatus for a wearable computer with natural user interface
An embodiment of a Wearable Computer with Natural User Interface apparatus includes a first portable unit for data gathering and communicating feedback and a second portable unit for processing the at least gathered data from the first unit. The first portable unit includes an eyeglass frame, at least one first optical unit disposed on the eyeglass frame for capturing at least one scene image corresponding to a field of view of a user, at least one second optical unit disposed on the eyeglass frame for capturing at least one eye image corresponding to at least a portion of at least one eye of the user, at least one microphone to allow the user to communicate via voice, at least one speaker to allow the user to receive feedback via voice, at least one visible light source to allow the user to receive feedback via light signals, at least one motion sensor to monitor the head movements of the user, and at least one first processor to at least receive data from the data gathering units in the first portable unit and at least manage the communication with the second portable unit. The second portable unit is in communication with the first portable unit and includes at least one second processor configured for receiving the at least data from the first processor and decoding a pre-defined command from the user and executing at least one command in response to the received command. At least one of the processors will determine a direction within the field of view to which the at least one eye is directed based upon the at least a history of one eye image, and generates a command or a subset of the at least one scene image based on the determined direction. At least one of the processors will provide a feedback to the user to acknowledge the user command received. In one embodiment, the Wearable Computer will function as a driver assistant and in another embodiment as a cameraman.
1. An apparatus of a wearable computer with natural user interface comprising: a first portable unit including:
an eyeglass frame to be worn by a user;
at least one scene camera disposed on the eyeglass frame, having a configurable field of view, for capturing at least one scene image corresponding to a field of view of the user;
at least one first eye tracking camera disposed on the eyeglass frame, having a configurable field of view, directed towards one eye of the user, for capturing at least a plurality of eye images corresponding to at least a portion of one eye of the user;
at least one accelerometer disposed on the eyeglass frame for tracking at least a head movement of the user and generating at least one signal corresponding to at least a duration of the head movement of the user;
at least one microphone disposed on the eyeglass frame for capturing at least a duration of the voice of the user;
at least one light emitter disposed on the eyeglass frame, for providing at least a light feedback to the user via light;
at least one bone conduction speaker disposed on the eyeglass frame for providing at least a voice feedback to the user via voice; and
at least one processor disposed on the eyeglass frame:
communicating with the at least one scene camera, the at least one first eye tracking camera, the at least one accelerometer, the at least one microphone, the at least one light emitter and the at least one conduction bone speaker,
receiving at least one signal from the at least one accelerometer, maintaining at least one history of the user's head movement based on at least a portion of the received signal from the at least one accelerometer,
receiving at least a plurality of eye images from the at least one first eye tracking camera, locating the pupil of the one eye of the user in the at least a plurality of the eye images received from the at least one first eye tracking camera and maintaining at least one history of the pupil location,
utilizing the at least one maintained history of the pupil location and the at least one maintained history of the head movement of the user to determine an area within a field of view of the user, wherein the size of the field of view of the determined area within a field of view of the user is between the size of the foveal view and the size of the peripheral view,
configuring the at least one scene camera to capture at least one scene image of the determined area within a field of view of the user,
receiving the at least one scene image corresponding to the determined area within a field of view of the user from the at least one scene camera, and transmitting the at least one scene image; and
a second portable unit including:
a smartphone, wherein the smartphone communicating at least with the at least one processor disposed in the first portable unit and the smartphone being carried by the user,
at least one user interface of a digital personal assistance app, wherein the app running on the smartphone and configured for communicating with the user at least via voice via the at least one microphone and the at least one conduction bone speaker, wherein the app further configured for:
receiving the at least one scene image from the at least one processor disposed in the first portable unit,
utilizing image processing to recognize as least one object in the at least one received scene image, and
taking at least one action based on the at least one recognized object in the at least one received scene image.
2. The apparatus in claim 1 , wherein the first portable unit further including at least one second eye tracking camera disposed on the eyeglass frame and directed towards the
other eye of the user, for capturing at least one eye image corresponding to at least a portion of the eye of the user and sending the at least one eye image to the at least one processor, wherein the at least one processor further configured for:
receiving at least one eye image from the at least one first eye tracking camera and one eye image from the at least one second eye tracking camera, and
estimating the user's distance from at least one point of the scene that the user's eyes directed at based on the at least one eye image from the at least one first eye tracking camera and the at least one eye image from the at least one second eye tracking camera.
3. The apparatus in claim 1 , wherein the at least one processor further configured for:
maintaining at least one history of the eye movement of the user based on the at least a plurality of the received eye images,
searching for a predetermined eye movement pattern within the at least one maintained history of eye movement, and
taking a predetermined action when a predetermined eye movement pattern is found.
4. The apparatus in claim 1 , wherein the digital personal assistance app configured to utilize object recognition to recognize at least one object in the at least one received scene image.
5. The apparatus in claim 1 , wherein the digital personal assistance app configured to utilize optical character recognition to identify at least one object in the at least one received scene image.
6. The apparatus in claim 1 , wherein the digital personal assistance app configured to utilize face recognition to identify at least one object in the at least one received scene image.
7. The apparatus in claim 1 , wherein taking the at least one action based on the at least one recognized object in the at least one received scene image includes anticipating an intended purpose of the user based on the at least one recognized object in the at least one received scene image.
8. The apparatus in claim 1 , wherein taking the at least one action based on the at least one recognized object in the at least one received scene image includes anticipating a need of the user based on the at least one recognized object in the at least one received scene image.
9. The apparatus in claim 1 , wherein taking the at least one action at least based on the at least one recognized object in the at least one received scene image includes providing the voice feedback to the user via the at least one conduction bone speaker.
10. The apparatus in claim 1 , wherein taking the at least one action at least based on the at least one recognized object in the at least one received scene image includes providing the light feedback to the user via the at least one light emitter.