Device interaction in augmented reality
An augmented reality (AR) device can access a library of applications or user interfaces (UIs) designed to control a set of devices. The AR device can determine which UI to present based on detection of a device to be controlled near the AR device. For example, a user wearing an AR device may look at a thermostat placed on a wall and a UI to control the thermostat may be presented to the user. The determination that the user is looking at the thermostat may be made by correlating the gaze tracking information of the user-facing camera with the location of the thermostat in an image captured by a world-facing camera. Determination of the location of the thermostat in the image can be performed using image recognition technology. The UI can be selected based on a database record pairing the UI with the thermostat.
1. A method comprising:
detecting, at a wearable computing device configured to be worn by a user, a plurality of devices in an image, the image captured by a camera that is configured to capture images of an environment of the wearable computing device, wherein the plurality of devices are physical objects in the environment;
determining a user selection of a target device of the plurality of devices, wherein the target device is of a different type than a first device of the plurality of devices;
establishing a wireless connection between the wearable computing device and the target device;
obtaining a status of the target device via the wireless connection;
selecting a control interface for the target device from a plurality of control interfaces, and the plurality of control interfaces are associated with different respective devices and includes a control interface for the first device, wherein each of the plurality of control interfaces includes a respective interactive element that controls an action of the associated device;
presenting, on a display of the wearable computing device, the status of the target device and the selected control interface, the selected control interface including an interactive element that controls an action of the target device, wherein the action of the target device is different from the action of the first device;
responsive to receiving a user interaction with the interactive element, transmitting instructions to the target device via the wireless connection to perform the action, wherein the target device performs the action in accordance with the instructions;
obtaining an updated status of the target device via the wireless connection; and
presenting on the display the updated status of the target device.
2. The method of claim 1 , wherein the determining the user selection of the target device comprises:
determining, via gaze detection, that the user is looking at the target device.
3. The method of claim 2 , further comprising:
determining, via gaze detection, that the user is no longer looking at the target device; and
ceasing the presentation of the selected control interface, based on the determination that the user is no longer looking at the target device and a predetermined time threshold.
4. The method of claim 1 , further comprising:
receiving a control input for the selected control interface via gaze detection.
5. The method of claim 1 , further comprising:
receiving a control input for the selected control interface via voice command.
6. The method of claim 1 , the method further comprising:
prior to the presenting of the selected control interface associated with the target device, presenting an augmented reality interface that includes a user interface for each of the plurality of devices in the image.
7. The method of claim 6 , wherein:
the user interface for device of the plurality of devices in the image, other than the user interface for the target device, remains visible while the selected control interface associated with the target device is presented.
8. The method of claim 6 , further comprising:
detecting a hand gesture; and
wherein:
the presenting of the selected control interface on the display is further in response to the detection of the hand gesture.
9. The method of claim 1 , wherein:
the display is an augmented reality display; and
the presenting of the selected control interface on the display comprises presenting at least a portion of the selected control interface superimposed over the target device.
10. The method of claim 1 , wherein:
the display is an augmented reality display; and
the presenting of the selected control interface on the display comprises presenting the selected control interface near to the target device.
11. A system comprising:
a display of a wearable computing device configured to be worn by a user;
a camera of the wearable computing device, the camera configured to capture images of an environment of the wearable computing device;
a memory storing instructions; and
one or more processors configured by the instructions to perform operations comprising:
detecting, at the wearable computing device, a plurality of devices in an image, the image captured by the camera, wherein the plurality of devices are physical objects in the environment,
determining a user selection of a target device of the plurality of devices, wherein the target device is of a different type than a first device of the plurality of devices,
obtaining a status of the target device via the wireless connection,
selecting a control interface for the target device from a plurality of control interfaces, and the plurality of control interfaces are associated with different respective devices and includes a control interface for the first device, wherein each of the plurality of control interfaces includes a respective interactive element that controls an action of the associated device,
presenting, on the display, the status of the target device and the selected control interface, the selected control interface including an interactive element that controls an action of the target device, wherein the action of the target device is different from the action of the first device,
responsive to receiving a user interaction with the interactive element, transmitting instructions to the target device via the wireless connection to perform the action, wherein the target device performs the action in accordance with the instructions,
obtaining an updated status of the target device via the wireless connection, and
presenting on the display the updated status of the target device.
12. The system of claim 10 , wherein the determining the user selection of the target device comprises:
determining, via gaze detection, that the user is looking at the target device.
13. The system of claim 12 , wherein the operations further comprise:
determining, via gaze detection, that the user is no longer looking at the target device; and
ceasing the presentation of the selected control interface, based on the determination that the user is no longer looking at the target device and a predetermined threshold.
14. The system of claim 11 , wherein the operations further comprise:
receiving a control input for the selected control interface via gaze detection.
15. The system of claim 11 , wherein the operations further comprise:
receiving a control input for the selected control interface via voice command.
16. The system of claim 11 , wherein the operations further comprise:
prior to the presenting of the selected control interface associated with the target device, presenting an augmented reality interface that includes a user interface for each of the plurality of devices in the image.
17. The system of claim 16 , wherein:
the user interface for each device of the plurality of devices in the image, other than the user interface for the target device, remains visible while the control interface associated with the target device is presented.
18. The system of claim 16 , wherein:
the operations further comprise:
detecting a hand gesture; and
the presenting of the selected control interface on the display is further in response to the detection of the hand gesture.
19. The system of claim 11 , wherein:
the display is an augmented reality display; and
the presenting of the selected control interface on the display comprises presenting at least a portion of the selected control interface superimposed over the target device.
20. A non-transitory machine-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
detecting, at a wearable computing device worn by a user, a plurality of devices in an image, the image captured by a camera that is configured to capture images of an environment of the wearable computing device, wherein the plurality of devices are physical objects in the environment;
determining a user selection of a target device of the plurality of devices, wherein the target device is of a different type than a first device of the plurality of devices;
establishing a wireless connection between the wearable computing device and the target device;
obtaining a status of the target device via the wireless connection;
selecting a control interface for the target device from a plurality of control interfaces, and the plurality of control interfaces are associated with different respective devices and includes a control interface for the first device, wherein each of the plurality of control interfaces includes a respective interactive element that controls an action of the associated device;
presenting, on a display of the wearable computing device, the status of the target device and the selected control interface, the selected control interface including an interactive element that controls an action of the target device, wherein the action of the target device is different from the action of the first device;
responsive to receiving a user interaction with the interactive element, transmitting instructions to the target device via the wireless connection to perform the action, wherein the target device performs the action in accordance with the instructions;
obtaining an updated status of the target device via the wireless connection; and
presenting on the display the updated status of the target device.