Hand-gesture activation of actionable items
In one implementation, a method of performing an action is performed at a device including an image sensor, one or more processors, and non-transitory memory. The method includes receiving, from the image sensor, one or more images of a physical environment. The method includes detecting, in the one or more images of the physical environment, one or more actionable items respectively associated with one or more actions. The method includes detecting, in the one or more images of the physical environment, a hand gesture indicating a particular actionable item. The method includes in response to detecting the hand gesture, performing an action associated with the particular actionable item.
1 . A method comprising:
at a device including an image sensor, one or more processors, and non-transitory memory:
receiving, from the image sensor, one or more images of a physical environment;
detecting, in the one or more images of the physical environment, one or more actionable items respectively associated with one or more actions, wherein detecting the one or more actionable items includes classifying the one or more actionable items to determine corresponding actions for the one or more actionable items, and wherein classifying the one or more actionable items includes performing computer vision on the one or more images of the physical environment;
detecting, in the one or more images of the physical environment, a first hand gesture selecting a particular actionable item of the one or more actionable items, the particular actionable item associated with a plurality of actions including a first action and a second action different from the first action;
in response to detecting the first hand gesture selecting the particular actionable item, without displaying a user interface element to perform the first action associated with the particular actionable item, performing the first action associated with the particular actionable item;
detecting, in the one or more images of the physical environment, a second hand gesture selecting the particular actionable item; and
in response to detecting the second hand gesture selecting the particular actionable item, without displaying a user interface element to perform the second action associated with the particular actionable item, performing the second action associated with the particular actionable item.
2 . The method of claim 1 , wherein detecting the one or more actionable items includes detecting machine-readable content.
3 . The method of claim 1 , wherein detecting the one or more actionable items includes detecting an object.
4 . The method of claim 3 , wherein performing the first action includes changing a state of the object.
5 . The method of claim 1 , wherein performing the first action includes playing audio based on the particular actionable item.
6 . The method of claim 5 , wherein the audio includes at least one of: a reading of the particular actionable item, a definition of the particular actionable item, or a translation of the particular actionable item.
7 . The method of claim 1 , wherein performing the first action includes initiating a phone call based on the particular actionable item.
8 . The method of claim 1 , wherein performing the first action includes storing, in the non-transitory memory, information based on the particular actionable item.
9 . The method of claim 1 , wherein performing the first action is further performed in response to a vocal command.
10 . The method of claim 9 , wherein performing the first action includes selecting, based on the vocal command, the first action from a plurality of actions associated with the particular actionable item.
11 . The method of claim 1 , wherein the device includes a communication interface, wherein the particular actionable item corresponds to another device, and wherein performing the first action includes transmitting data to the other device via the communication interface.
12 . The method of claim 11 , wherein the data indicates a request to change a state of the other device.
13 . The method of claim 1 , wherein the first hand gesture is a swipe hand gesture.
14 . The method of claim 1 , wherein the first hand gesture is a circling hand gesture.
15 . The method of claim 1 , wherein the first hand gesture is a tap hand gesture and the second hand gesture is a double-tap hand gesture.
16 . The method of claim 1 , wherein the first hand gesture is a circle hand gesture in which a finger contacts a thumb to form a circle.
17 . A device comprising:
an image sensor;
a non-transitory memory; and
one or more processors to:
receive, from the image sensor, one or more images of a physical environment;
detect, in the one or more images of the physical environment, one or more actionable items respectively associated with one or more actions, wherein detecting the one or more actionable items includes classifying the one or more actionable items to determine corresponding actions for the one or more actionable items, and wherein classifying the one or more actionable items includes performing computer vision on the one or more images of the physical environment;
detect, in the one or more images of the physical environment, a first hand gesture selecting a particular actionable item of the one or more actionable items, the particular actionable item associated with a plurality of actions including a first action and a second action different from the first action;
in response to detecting the first hand gesture selecting the particular actionable item, without displaying a user interface element to perform a the first action associated with the particular actionable item, perform the first action associated with the particular actionable item;
detect, in the one or more images of the physical environment, a second hand gesture selecting the particular actionable item; and
in response to detecting the second hand gesture selecting the particular actionable item, without displaying a user interface element to perform the second action associated with the particular actionable item, perform the second action associated with the particular actionable item.
18 . The device of claim 17 , wherein the one or more processors are to detect the one or more actionable items by detecting machine-readable content or an object.
19 . The device of claim 17 , wherein the device does not include a display.
20 . A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device including an image sensor cause the device to:
receive, from the image sensor, one or more images of a physical environment;
detect, in the one or more images of the physical environment, one or more actionable items respectively associated with one or more actions, wherein detecting the one or more actionable items includes classifying the one or more actionable items to determine corresponding actions for the one or more actionable items, and wherein classifying the one or more actionable items includes performing computer vision on the one or more images of the physical environment;
detect, in the one or more images of the physical environment, a first hand gesture selecting a particular actionable item of the one or more actionable items, the particular actionable item associated with a plurality of actions including a first action and a second action different from the first action;
in response to detecting the first hand gesture selecting the particular actionable item, without displaying a user interface element to perform a the first action associated with the particular actionable item, perform the first action associated with the particular actionable item;
detect, in the one or more images of the physical environment, a second hand gesture selecting the particular actionable item; and
in response to detecting the second hand gesture selecting the particular actionable item, without displaying a user interface element to perform the second action associated with the particular actionable item, perform the second action associated with the particular actionable item.