Augmented reality with motion sensing
View Patent ↗The technology relates to a motion sensory and imaging device capable of acquiring imaging information of the scene and providing at least a near real time pass-through of imaging information to a user. The sensory and imaging device can be used stand-alone or coupled to a wearable or portable device to create a wearable sensory system capable of presenting to the wearer the imaging information augmented with virtualized or created presentations of information.
1. A device, including:
a plurality of imaging sensors, including a first set of infrared pixels sensitive to infrared light and a second set of red-green-blue pixels sensitive to visible light; wherein imaging sensors are arranged to provide stereoscopic imaging information for a scene in front of a user wearing a coupled wearable or portable device that blocks the scene from being viewed including at least one hand, said at least one hand belonging to the user wearing the coupled wearable or portable device and moving within the scene, and wherein said at least one hand is blocked from being viewed by the user;
one or more illumination sources arranged to direct illumination towards the at least one hand; and
a controller coupled to the imaging sensors and illumination sources to control operation thereof, including:
operating the imaging sensors and illumination sources to acquire imaging information of the scene, including capturing imaging information for at least one hand within view of the imaging sensors; wherein at least part of the at least one hand is illuminated by the one or more illumination sources;
separating imaging information received from infrared pixels sensitive to infrared light from imaging information received from red-green-blue pixels sensitive to visible light;
processing imaging information from infrared pixels to determine gesture information of a user indicating a command to a machine under control;
processing imaging information from red-green-blue pixels to be provided as a live video feed via a presentation interface; and
providing at least: i) gestural information of the at least one hand determined using the imaging information from infrared pixels for the at least one hand; and ii) a display of the live video feed including imaging information of the scene for presentation to a user via a coupled wearable or portable device worn by a user; wherein the user wearing the coupled wearable or portable device is devoid of visual information from the scene surrounding the user other than from the live video feed.
2. The device of claim 1 , wherein the processing imaging information from infrared pixels further includes:
extracting fine features of a real world space using infrared pixels that capture infrared components of illumination in the scene.
3. The device of claim 2 , wherein fine features of the real world space include surface texture of the real world space.
4. The device of claim 2 , wherein fine features of the real world space include edges of the real world space.
5. The device of claim 2 , wherein fine features of the real world space include curvatures of the real world space.
6. The device of claim 2 , wherein fine features of the real world space include surface texture of objects in the real world space.
7. The device of claim 2 , wherein fine features of the real world space include edges of objects in the real world space.
8. The device of claim 2 , wherein fine features of the real world space include curvatures of objects in the real world space.
9. The device of claim 1 ,
wherein the controller further provides:
determining ambient lighting conditions;
adjusting display of output based upon the ambient lighting conditions determined, including:
dynamically switching display between video information captured using red-green-blue pixels sensitive to visible light and video information captured using infrared pixels sensitive to infrared light for display depending upon ambient conditions.
10. The device of claim 1 , further including:
a motion sensor; and wherein the controller further provides:
determining difference information between a first and a second positional information from the motion sensor; and
computing movement information for the device with respect to a fixed point based upon the difference information.
11. The device of claim 1 , further including:
one or more fasteners that fasten the imaging sensors and the illumination sources to a mounting surface in a wearable presentation device.
12. The device of claim 1 , further including:
one or more fasteners that fasten the imaging sensors and the illumination sources to a cavity in a wearable presentation device.
13. The device of claim 1 , further including:
one or more fasteners that fasten the imaging sensors and the illumination sources to a mounting surface in a portable presentation device.
14. The device of claim 1 , further including:
one or more fasteners that fasten the imaging sensors and the illumination sources to a cavity in a portable presentation device.
15. The device of claim 1 , further including the display of the live video feed including images of the hand grasping a real object.
16. The device of claim 15 , further including the display of the live video feed including images of the hand grasping a virtual object.
17. The device of claim 1 , further including the plurality of image sensors being disposed co-planar on a head mounted display.
18. The device of claim 17 , the head mounted display including an optical assembly that displays a surrounding environment or a virtual environment to the user.
19. The device of claim 1 , further including detecting a position and shape of a user's hand and projecting onto a display of a head-mounted device; wherein the user is presented with gestures in the display of the live video feed that reflecting hand motions of the user captured by the infrared pixels and interactively control objects in a virtual environment.