Intuitive computing methods and systems
A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.
1. A method performed using a system comprising a local device with one or more remote servers, the system including plural processors, the local device including a camera that captures imagery from a user's physical environment and a microphone that captures user speech, the method comprising the acts:
capturing user speech with the microphone, the captured speech including a first portion followed by a second portion;
recognizing that the first portion of the captured speech includes a cueing expression, said recognizing employing one of said processors;
in response to recognizing of the cueing expression, switching the system from a lower activity state to a heightened alert state;
in said heightened alert state sending captured speech data, including data representing said second portion of user speech, to a first remote server for recognition, the second portion of user speech indicating that the user desires the system to perform a camera-related operation;
responsive to recognition by the first remote server of said data representing the second portion of user speech, the system undertaking an image recognition task at a second remote server, using imagery captured by the local device camera;
in response to first response data indicating failure of the second remote server to satisfactorily complete said image recognition task, the local device providing verbal feedback to the user, said verbal feedback indicating said failure and requesting additional information;
following said failure, submitting additional information from the local device for use in said task;
after submitting said additional information from the local device, receiving second response data indicating successful completion of the image recognition task and including associated result information; and
presenting said result information to the user using said local device;
wherein initial failure of the second remote server to complete the image recognition task is indicated to the user by verbal feedback that requests additional information, and wherein submission of the additional information after said verbal feedback is followed by receipt of result information associated with successful completion of the image recognition task.
2. The method of claim 1 in which the first remote server and the second remote server comprise a single remote server.
3. The method of claim 1 in which said image recognition task comprises reading a barcode depicted in the captured imagery.
4. The method of claim 3 in which said second portion of user speech does not explicitly declare that the user desires the system to perform a barcode reading operation.
5. The method of claim 1 in which said image recognition task comprises optical character recognition of text depicted in the captured imagery.
6. The method of claim 1 in which a noun recognized in the second portion of user speech indicates which image recognition task, from among plural image recognition tasks, is to be undertaken by the system.
7. The method of claim 6 in which said plural image-recognition tasks include at least barcode reading and optical character recognition.
8. A method performed using a system comprising a local device with one or more remote servers, the system including plural processors, the local device including a camera that captures imagery from a user's physical environment and a microphone that captures user speech, the method comprising the acts:
capturing user speech with the microphone, the captured speech including a cueing expression and a following portion;
recognizing that the captured speech includes said cueing expression, said recognizing employing one of said processors;
in response to recognizing the cueing expression, switching the system from a lower activity state to a heightened alert state;
in said heightened alert state sending captured speech data, including data representing said following portion of user speech, to a first remote server for recognition, the following portion of user speech indicating that the user desires the system to perform a camera-related operation;
responsive to recognition by the first remote server of said data representing the following portion of user speech, the system undertaking an image recognition task using imagery captured by the local device camera;
receiving initial result information associated with a result of said image recognition task, and presenting said initial result information to the user using said local device;
after presenting said initial result information to the user, capturing further user speech;
sending further user speech data representing said captured further user speech to a remote computer; and
after said sending of the further user speech data, receiving further result information, and presenting said further result information to the user using said local device;
wherein the system operates in iterative fashion, responding to user speech with initial result information associated with an image processing task, and then responding to further user speech with further result information.
9. The method of claim 8 in which the first remote server and the remote computer comprise a single remote server.
10. The method of claim 8 in which said image recognition task comprises reading a barcode depicted in the captured imagery.
11. The method of claim 10 in which said following portion of user speech does not explicitly declare that the user desires the system to perform a barcode reading operation.
12. The method of claim 8 in which said image recognition task comprises optical character recognition of text depicted in the captured imagery.
13. The method of claim 8 in which a noun recognized in the following portion of user speech indicates which image recognition task, from among plural image recognition tasks, is to be undertaken by the system.
14. The method of claim 13 in which said plural image recognition tasks includes at least barcode reading and optical character recognition.