Methods and systems for creating virtual and augmented reality
View Patent ↗Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.
1. An augmented reality display system, comprising:
an image capturing device of the augmented reality display system to capture at least one image, wherein
the image capturing device comprises one or more image capturing sensors,
at least a portion of the at least one image is perceived within a field of view of a user, and
the at least one image captures at least one gesture that is created by the user and interacts with virtual content projected by the augmented reality display system to the user; and
a processor coupled directly with no intervening elements or indirectly with one or more intervening elements to the image capturing device to recognize the at least one gesture as at least one recognized gesture, the processor configured to recognize the at least one gesture as the at least one recognized gesture is further configured to:
identify a plurality of candidate gestures and a plurality of computational utilization or expense requirements for gesture recognition of the at least one gesture;
determine whether the at least one image includes one or more identifiable depth points at least by performing a line search for the at least one image with one or more lines or line segments;
determine an order of processing in which a plurality of analysis nodes is executed to perform respective gesture identification processes on the at least one gesture with respect to the plurality of candidate gestures based at least in part upon the plurality of computational resource utilization or expense requirements;
during one or more earlier stages in the order of processing, generate one or more reduced sets of candidate gestures from the plurality of candidate gestures at least by executing one or more first gesture identification processes of the respective gesture identification processes that analyze the at least one image with a first analysis node to remove one or more candidate gestures from the plurality of candidate gestures based at least in part upon a first computational resource utilization or expense requirement of the plurality of computational resource utilization or expense requirements; and
during one or more later stages in the order of processing, determine and recognize the at least one gesture from the one or more reduced sets of candidate gestures as the at least one recognized gesture at least by analyzing the at least one image based at least in part on the one or more reduced sets of candidate gestures with a second analysis node corresponding to a second computational expense criterion and at least by executing one or more second gesture identification processes on the at least one gesture and the one or more reduced sets of candidate gestures, wherein
the one or more second gesture identification processes consume a larger amount of processing power than the one or more first gesture identification processes; and
the processor further configured to determine a user input based at least in part on the at least one recognized gesture.
2. The augmented reality display system of claim 1 , wherein the processor is configured to generate a scoring value for a set of points identified for the at least one gesture based at least in part on comparison between the set of points and predetermined gestures and to recognize the at least one gesture when the scoring value exceeds a threshold value.
3. The augmented reality display system of claim 1 , further comprising a database to store predetermined gestures, wherein the computational resource utilization or expense requirement includes reducing or minimizing computational resource utilization for the gesture recognition for the at least one gesture, the first computational resource or expense requirement corresponds to a relatively lower computation resource utilization, and the second computational resource or expense requirement corresponds to a relatively higher computational resource utilization when compared to the first computational resource or expense requirement.
4. The augmented reality display system of claim 3 , further comprising a networked memory to access the database of predetermined gestures.
5. The augmented reality display system of claim 1 , wherein the processor is further configured to recognize the at least one gesture that comprises a hand gesture or motion or a finger gesture or a finger motion.
6. The augmented reality display system of claim 1 , wherein the augmented reality display system comprises a user wearable apparatus to display a virtual world as well as at least a portion of a physical environment in which the user is located.
7. The augmented reality display system of claim 1 , where the processor is further configured to recognize the at least one gesture that comprises an inter-finger interaction.
8. The augmented reality display system of claim 1 , wherein the processor is further configured to recognize the at least one gesture comprising at least one of inter-finger interactions, pointing, tapping, or rubbing.
9. The augmented reality display system of claim 1 , further comprising a spatial light modulator that is coupled directly without intervening elements or indirectly with one or more elements to the processor, and the processor controls the spatial light modulator in a manner such that one or more virtual objects are displayed to the user based at least in part on the user input.
10. The augmented reality display system of claim 9 , further comprising a virtual user interface to receive the user input or a user interaction with the virtual user interface or with the one or more virtual objects.
11. A method for determining user input, comprising:
capturing an image corresponding to a field of view of a user through an augmented reality system, wherein
the image comprises a gesture image of at least one gesture that is created by the user and interacts with virtual content projected by the augmented reality system to the user, and
at least a portion of the image is perceived by the user within the field of view provided by the augmented reality system;
identifying a plurality of candidate gestures and a plurality of computational utilization or expense requirements for gesture recognition of the at least one gesture;
determining whether the image includes one or more identifiable depth points at least by performing a line search for the image with one or more lines or line segments;
determining an order of processing in which a plurality of analysis nodes is executed to perform respective gesture identification processes on the at least one gesture with respect to the plurality of candidate gestures based at least in part upon a plurality of computational resource utilization or expense requirements;
during one or more earlier stages in the order of processing, generating one or more reduced sets of candidate gestures from the plurality of candidate gestures at least by executing one or more first gesture identification processes of the respective gesture identification processes that analyze the at least one image with a first analysis node to remove one or more candidate gestures from the plurality of candidate gestures based at least in part upon a first computational resource utilization or expense requirement of the plurality of computational resource utilization or expense requirements;
during one or more later stages in the order of processing, determining and recognizing the at least one gesture from the one or more reduced sets of candidate gestures as the at least one recognized gesture at least by analyzing the at least one image based at least in part on the one or more reduced sets of candidate gestures with a second analysis node corresponding to a second computational resource or expense requirement and at least by executing one or more second gesture identification processes on the at least one gesture and the one or more reduced sets of candidate gestures, wherein
the one or more second gesture identification processes consume a larger amount of processing power than the one or more first gesture identification processes; and
determining a user input based in part or in whole upon the at least one recognized gesture.
12. The method of claim 11 , further comprising generating a scoring value for a set of points for the at least one gesture based in part or in whole on results of comparing the set of points to a first set of points associated with a database including predetermined gestures.
13. The method of claim 12 , further comprising recognizing the at least one gesture when the scoring value exceeds a threshold value.
14. The method of claim 11 , further comprising overlaying a virtual world with at least a portion of a physical environment in which the user is located.
15. The method of claim 14 , further comprising accessing a networked memory to access a database including predetermined gestures.
16. The method of claim 11 , determining the user input further comprising recognizing the at least one gesture that comprises a hand gesture, a hand motion, a finger gesture, or a finger motion.
17. The method of claim 11 , further comprising capturing and recognizing a sensory input from the user or from a physical environment in which the user is located.
18. The method of claim 11 , determining the user input further comprising recognizing the at least one gesture that comprises an inter-finger interaction.
19. The method of claim 11 , determining the user input further comprising recognizing the at least one gesture that comprises at least one of inter-finger interactions, pointing, tapping, or rubbing.
20. The method of claim 11 , further comprising displaying one or more virtual objects to the user based at least in part on the user input.
21. The method of claim 20 , further comprising recognizing a user interaction with the one or more virtual objects or with a virtual world in which the one or more virtual objects are displayed in a virtual user interface based in part or in whole upon the at least one gesture.
22. The method of claim 11 , further comprising:
generating, at a processor of the augmented reality system, a scoring value for a set of points identified for the at least one gesture based at least in part on comparison between the set of points and predetermined gestures; and
recognizing the at least one gesture as the at least one recognized gesture when the scoring value exceeds a threshold value.