IP Library › Granted Patent US 8,489,115
Granted Patent B2
US 8,489,115 · App. 13/466,803 · Granted Jul 16, 2013

Sensor-based mobile search, related methods and systems

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 8,489,115
App. No.
13/466,803
Filed
May 8, 2012
Granted
Jul 16, 2013
Kind
B2
Art Unit
2645
USPC
455/456.1
Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims (40)

1. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus being other than speech;

processing the received first and second data to generate output data indicating a possible identity of said unknown subject, said processing being performed at least in part by a processing system in the smartphone that is configured to perform part or all of said processing;

wherein said processing includes consulting disambiguation information recalled from a data store, the disambiguation data having been earlier obtained by analysis of a corpus of content information posted by plural third parties to one or more public internet sites.

2. The method of claim 1 that additionally includes analyzing said corpus of content to derive the disambiguation data.

3. The method of claim 2 in which the analyzing includes analyzing visual content.

4. The method of claim 2 in which the analyzing includes analyzing textual metadata accompanying visual content.

5. The method of claim 1 in which the sensor comprises a microphone.

6. The method of claim 1 in which the sensor comprises a thermometer.

7. The method of claim 1 in which the sensor comprises a location sensing system.

8. The method of claim 1 in which the sensor comprises a chemical sensor.

9. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

processing the first data to generate object identification data indicating a possible identity of an object depicted in the imagery, said object being different than said unknown subject; and

recognizing the unknown subject, said recognizing being based at least in part both on said object identification data, and also on co-occurrence information involving said object;

wherein said co-occurrence information was earlier obtained by analysis of a corpus of content information posted by plural third parties to one or more public internet sites.

10. The method of claim 9 that additionally includes analyzing said corpus of content to derive the co-occurrence information.

11. The method of claim 10 in which the analyzing includes analyzing visual content.

12. The method of claim 10 in which the analyzing includes analyzing textual metadata accompanying visual content.

13. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

receiving second data corresponding to non-speech audio stimulus captured by a microphone of the smartphone;

processing the received first and second data to generate output data indicating a possible identity of said unknown subject, said processing being performed at least in part by a processing system in the smartphone that is configured to perform part or all of said processing;

wherein said processing includes computing a score for each of plural different candidate identifications of said unknown subject, at least one of said scores reflecting an uncertainty factor corresponding to the audio stimulus.

14. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

deriving recognition features from the imagery, said deriving being performed by a processing system in the smartphone configured to perform said deriving;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus comprising audio or temperature;

from a set of reference recognition features associated with a first set of visual subjects, identifying a smaller subset of recognition features associated with a second, smaller set of visual subjects, said identifying being based, at least in part, on the second data; and

identifying the unknown subject from among said second set of subjects, by correspondence between the derived recognition features and recognition features in said subset.

15. The method of claim 14 wherein said recognition features comprise scale invariant features.

16. The method of claim 14 wherein said non-image stimulus comprises non-speech audio.

17. The method of claim 14 wherein said non-image stimulus comprises audio from a source different than said subject.

18. The method of claim 14 wherein said non-image stimulus comprises temperature.

19. A non-transitory computer readable storage medium containing software instructions that, when executed by a processor of a camera-equipped smartphone device, cause the smartphone device to perform acts including:

receiving first data corresponding to imagery captured by the camera of a smartphone, the imagery depicting an unknown subject;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus being other than speech;

processing the received first and second data to generate output data indicating a possible identity of said unknown subject;

wherein said processing includes consulting disambiguation information recalled from a data store, the disambiguation data having been earlier obtained by analysis of a corpus of content information posted by plural third parties to one or more public internet sites.