Touch and social cues as inputs into a computer
A system for automatically displaying virtual objects within a mixed reality environment is described. In some embodiments, a see-through head-mounted display device (HMD) identifies a real object (e.g., a person or book) within a field of view of the HMD, detects one or more interactions associated with real object, and automatically displays virtual objects associated with the real object if the one or more interactions involve touching or satisfy one or more social rules stored in a social rules database. The one or more social rules may be used to infer a particular social relationship by considering the distance to another person, the type of environment (e.g., at home or work), and particular physical interactions (e.g., handshakes or hugs). The virtual objects displayed on the HMD may depend on the particular social relationship inferred (e.g., a friend or acquaintance).
1. A method, comprising:
identifying, by a first image processing operation, a particular person within a field of view of a head mounted display (HMD) device;
detecting, by a second image processing operation, that a user of the HMD device has performed a gesture directed towards the particular person, wherein the gesture includes the user of the HMD device touching the particular person;
determining whether the gesture satisfies one or more social rules, wherein the one or more social rules correspond to a social relationship with the particular person based on an environment surrounding the user of the HMD device and the particular person, a distance between the user of the HMD device and the particular person, a vocal expression or a facial expression of the particular person towards the user of the HMD device, and a period of time coinciding with an electronically scheduled meeting between the user of the HMD device and the particular person;
acquiring virtual data associated with an application for an augmented reality environment to be displayed in response to the one or more social rules being satisfied, wherein the application corresponds to the electronically scheduled meeting between the user of the HMD device and the particular person; and
displaying the application using the HMD device.
2. The method of claim 1 ,
wherein the application corresponds to an electronic calendar.
3. The method of claim 1 , wherein:
the gesture is one of a handshake gesture, a hug gesture, or a high-five gesture.
4. The method of claim 1 , wherein:
the detecting that the user of the HMD device has touched the particular person includes performing skeletal tracking of the particular person.
5. The method of claim 1 , wherein:
the acquiring the virtual data includes acquiring a plurality of virtual objects displayed within the augmented reality environment.
6. The method of claim 5 , further comprising:
transmitting other virtual data associated with a second augmented reality environment displayed to the user of the HMD device to a mobile device displaying the plurality of virtual objects to the particular person.
7. The method of claim 6 , wherein:
the transmitting the other virtual data to the mobile device includes transmitting, to the mobile device, a set of virtual objects displayed using the HMD device at a point in time coinciding with an electronically scheduled meeting between the user of the HMD device and the particular person.
8. A head mounted display (HMD) device, comprising:
a display;
a memory storing instructions; and
one or more processors communicatively coupled with the display and the memory and configured to:
identify, by a first image processing operation, a particular person within a field of view of the display;
detect, by a second image processing operation, that a user of the HMD device has performed a gesture directed towards the particular person, wherein the gesture includes the user of the HMD device touching the particular person;
determine whether the gesture satisfies one or more social rules, wherein the one or more social rules correspond to a social relationship with the particular person based on an environment surrounding the user of the HMD device and the particular person, a distance between the user of the HMD device and the particular person, a vocal expression or a facial expression of the particular person towards the user of the HMD device, and a period of time coinciding with an electronically scheduled meeting between the user of the HMD device and the particular person; and
acquire virtual data associated with an application for an augmented reality environment to be displayed in response to the one or more social rules being satisfied, wherein the application corresponds to the electronically scheduled meeting between the user of the HMD device and the particular person,
wherein the display is configured to display the application subsequent to acquisition of the virtual data.
9. The HMD device of claim 8 ,
wherein the application corresponds to an electronic calendar.
10. The HMD device of claim 8 , wherein the gesture is one of a handshake gesture, a hug gesture, or a high-five gesture.
11. The HMD device of claim 8 , wherein the one or more processors is further configured to:
perform skeletal tracking of the particular person; and
detect that the user of the HMD device has touched the particular person based on the skeletal tracking.
12. The HMD device of claim 8 , wherein the one or more processors is further configured to:
transmit other virtual data associated with a second augmented reality environment displayed to the user of the HMD device to a mobile device displaying the virtual data to the particular person.
13. The HMD device of claim 12 , wherein the one or more processors is further configured to transmit, to the mobile device, a set of virtual objects displayed using the HMD device.
14. The HMD device of claim 8 , wherein the display comprises a see-through display.
15. One or more non-transitory storage devices containing processor readable code for programming one or more processors, comprising:
processor readable code configured to identify, by a first image processing operation, a particular person within a field of view of a display of a head mounted display (HMD) device;
processor readable code configured to detect, by a second image processing operation, that a user of the HMD device has performed a gesture directed towards the particular person, wherein the gesture includes the user of the HMD device touching the particular person;
processor readable code configured to determine whether the gesture satisfies one or more social rules, wherein the one or more social rules correspond to a social relationship with the particular person based on an environment surrounding the user of the HMD device and the particular person, a distance between the user of the HMD device and the particular person, a vocal expression or a facial expression of the particular person towards the user of the HMD device, and a period of time coinciding with an electronically scheduled meeting between the user of the HMD device and the particular person;
processor readable code configured to acquire virtual data associated with an application for an augmented reality environment to be displayed in response to the one or more social rules being satisfied, wherein the application corresponds to the electronically scheduled meeting between the user of the HMD device and the particular person; and
processor readable code configured to display the virtual data application using the HMD device.