Systems, methods, and apparatus for providing image shortcuts for an assistant application
Methods, apparatus, systems, and computer-readable media are set forth for generating and/or utilizing image shortcuts that cause one or more corresponding computer actions to be performed in response to determining that one or more features are present in image(s) from a camera of a computing device of a user (e.g., present in a real-time image feed from the camera). An image shortcut can be generated in response to user interface input, such as a spoken command. For example, the user interface input can direct the automated assistant to perform one or more actions in response to object(s) having certain feature(s) being present in a field of view of the camera. Subsequently, when the user directs their camera at object(s) having such feature(s), the assistant application can cause the action(s) to be automatically performed. For example, the assistant application can cause data to be presented and/or can control a remote device in accordance with the image shortcut.
1. A computing device, comprising:
a microphone;
a display device;
a front facing camera that faces in the same direction as the display device;
a speaker;
one or more processors in communication with the front facing camera, the microphone, the display device, and the speaker; and
memory configured to store instructions that, when executed by the one or more processors, cause one or more of the processors to:
process one or more images, generated by the front facing camera, to determine that one or more of the images include:
an object, and
a face of a particular user;
responsive to determining that the one or more images include the object and the face of the particular user:
invoke an image shortcut setting,
wherein the image shortcut setting is generated in response to one or more previous inputs from the particular user to an automated assistant application of an automated assistant, and
wherein invoking the image shortcut setting causes the automated assistant to perform one or more computer actions.
2. The computing device of claim 1 , wherein the one or more computer actions performed by the automated assistant comprise transmitting one or more commands that cause the state of an Internet of Things device to be altered.
3. The computing device of claim 1 , wherein the one or more computer actions performed by the automated assistant comprise transmitting a generated query, receiving responsive data in response to transmitting the generated query, and causing at least a portion of the responsive data to be rendered at the computing device.
4. The computing device of claim 1 , wherein the one or more computer actions performed by the automated assistant comprise transmitting an electronic communication to a contact of the particular user.
5. The computing device of claim 1 , wherein one or more of the processors are further to:
prior to processing the one or more images, generate the image shortcut setting, wherein in generating the image shortcut setting one or more of the processors are to:
receive a spoken utterance from the particular user,
determine, in response to receiving the spoken utterance, that natural language content of the spoken utterance includes a term that identifies the object and further includes a request to create the image shortcut setting corresponding to the object, and
generate, based on determining that the spoken utterance includes the term and the request, the image shortcut setting.
6. The computing device of claim 1 , wherein the front facing camera provides a real-time image feed that is rendered at the display device.
7. The computing device of claim 1 , wherein in invoking the image shortcut setting, one or more of the processors are to invoke the image shortcut setting further based on a time or date, corresponding to capturing of the one or more images, matching a temporal condition for invoking the image shortcut setting.
8. A method implemented by one or more processors, the method comprising:
processing one or more images, generated by a camera of a client device, to determine that one or more of the images include:
an object, and
a face of a particular user;
responsive to determining that the one or more images include the object and the face of the particular user:
invoking an image shortcut setting,
wherein the image shortcut setting is generated in response to one or more previous inputs from the particular user to an automated assistant application of an automated assistant, and
wherein invoking the image shortcut setting causes the automated assistant to perform one or more computer actions.
9. The method of claim 8 , wherein the one or more computer actions performed by the automated assistant comprise transmitting one or more commands that cause the state of an Internet of Things device to be altered.
10. The method of claim 8 , wherein the one or more computer actions performed by the automated assistant comprise transmitting a generated query, receiving responsive data in response to transmitting the generated query, and causing at least a portion of the responsive data to be rendered at the computing device.
11. The method of claim 8 , wherein the one or more computer actions performed by the automated assistant comprise transmitting an electronic communication to a contact of the particular user.
12. The method of claim 8 , wherein the method further comprises:
prior to processing the one or more images, generating the image shortcut setting, wherein generating the image shortcut setting includes:
receiving a spoken utterance from the user,
determining, in response to receiving the spoken utterance, that natural language content of the spoken utterance includes a term that identifies the object and further includes a request to create the image shortcut setting corresponding to the object, and
generating, based on determining that the spoken utterance includes the term and the request, the image shortcut setting.
13. The method of claim 8 , wherein the camera is a front facing camera that faces the same direction as a display of the client device.
14. The method of claim 8 , further comprising:
determining that a time or date, corresponding to capturing of the one or more images, matches a temporal condition for invoking the image shortcut setting;
wherein invoking the image shortcut setting is further based on the time or date matching the temporal condition for invoking the image shortcut setting.
15. At least one non-transitory computer readable medium storing instructions that, when executed by processor, cause the processor to:
process one or more images, generated by a camera of a client device, to determine that one or more of the images include:
an object, and
a face of a particular user;
responsive to determining that the one or more images include the object and the face of the particular user:
invoke an image shortcut setting,
wherein the image shortcut setting is generated in response to one or more previous inputs from the particular user to an automated assistant application of an automated assistant, and
wherein invoking the image shortcut setting causes the automated assistant to perform one or more computer actions.