Systems and methods for image-feature-based recognition
Methods and systems are described herein that allow a user to capture a single image snapshot from video, print, or the world around him or her, and obtain additional information relating to the media itself or items of interest displayed in the snapshot. A fingerprint of the snapshot is used as a query and transmitted to the server. Image Feature-Based Recognition, as described herein, uses a feature index to identify a smaller set of candidate matches from a larger database of images based on the fingerprint. Novel methods and systems using a distance metric and a radical hash table design exploit probabilistic effects and allow distinct image features to be preferred over redundant ones, allowing only the more distinctive data points to remain resident within the index, yielding a lean index that can be quickly used in the identification process.
1. A server for determining information relating to images captured by mobile devices, the server comprising a processor for executing one or more modules comprising:
a query receiving module for receiving, from a first mobile device, a first query including a first plurality of query image feature descriptors;
a candidate feature descriptors retrieval module for
determining, for at least one of the first query image feature descriptors, a plurality of first addresses using a plurality of functions, and
determining, within an address space associated with the plurality of first addresses at a reference image database, a number of duplicate reference image feature descriptors, wherein a plurality of duplicates of at least one reference image feature descriptor is mapped using the plurality of functions to at least one second address at the reference image database;
a candidate images retrieval module for selecting one or more reference image feature descriptors based on the determined number of duplicate reference image feature descriptors;
wherein, based on the selected reference image feature descriptors, the server is further configured to perform the steps of:
determining information associated with the selected reference image feature descriptors;
providing the information associated with the selected reference image feature descriptors to the first mobile device;
receiving a second query from a second mobile device, the second query including the same plurality of query image feature descriptors as in the first query; and
providing the same information associated with the selected reference image feature descriptors to the second mobile device.
2. The server of claim 1 , wherein the first query comprises the most stable image out of multiple images captured by the first mobile device, wherein the server determines an image is stable when it detects little or no movement.
3. The server of claim 1 , wherein the first query comprises an image taken some amount of time prior to a snapshot being taken, when there is a delay between what is displayed on the screen and when the snapshot is taken after the user initiates the taking of the snapshot.
4. The server of claim 1 , wherein the first plurality of query image feature descriptors are generated based on salient points detected in a portion or the entirety of the query image.
5. The server of claim 1 , wherein the processor is further configured to execute a hash table management module for:
providing, for each of the plurality of first addresses, one or more references to one or more of the reference image feature descriptors based on a relationship between each of the plurality of first addresses and the least one second address;
and wherein the number of duplicate reference image feature descriptors is determined based on the relationship.
6. The server of claim 1 , wherein the processor is further figured to execute the candidate images retrieval module for:
ranking one or more sets of reference image feature descriptors, each set being associated with a combination of the determined numbers of duplicate reference image feature descriptors; and
selecting the set of reference image feature descriptors based on a result of the ranking.
7. The sever of claim 1 , wherein the processor is further configured to execute the candidate images retrieval module for:
excluding, from the selection of the one or more reference image feature descriptors, reference image feature descriptors associated with a calculated number of duplicate reference image feature descriptors that is below a predefined threshold.
8. The server of claim 1 , wherein the reference image feature descriptors are stored based on a hash table including a plurality of hash buckets, each hash bucket being associated with an address within the address space; wherein the plurality of functions are configured to map one or more feature descriptors to one or more hash buckets of a set of hash buckets.
9. The server of claim 1 , wherein the processor is further configured to perform the steps of:
receiving data associated with media material through an interface; and
providing the data to the mobile device as part of the information associated with the selected image feature descriptors.
10. A computer-implemented method for determining information relating to images captured by mobile devices, the method comprising:
receiving, from the first mobile device, a first query including a first plurality of query image feature descriptors;
determining, within an address space associated with a plurality of first addresses determined from at least one of the first query image feature descriptors, a number of duplicate reference image feature descriptors, wherein a plurality of duplicates of at least one reference image feature descriptor is mapped using a plurality of functions to at least a second address;
selecting one or more reference image feature descriptors based on the determined number of duplicate reference image feature descriptors;
providing the information associated with the selected reference image feature descriptors to the mobile device;
receiving a second query from a second mobile device, the second query including the same plurality of query image feature descriptors as in the first query; and
providing the same information associated with the selected reference image feature descriptors to the second mobile device.
11. The method of claim 10 , wherein the first query comprises the most stable image out of multiple images captured by the first mobile device, wherein the method determines an image is stable when it detects little or no movement.
12. The method of claim 10 , wherein the first query comprises an image taken some amount of time prior to a snapshot being taken, when there is a delay between what is displayed on the screen and when the snapshot is taken after the user initiates the taking of the snapshot.
13. The method of claim 10 , wherein the first plurality of query image feature descriptors are generated based on salient points detected in a portion or the entirety of the query image.
14. The method of claim 10 , further comprising:
providing, for each of the plurality of first addresses, one or more references to one or more of the reference image feature descriptors based on a relationship between each of the plurality of first addresses and the least one second address;
and wherein the number of duplicate reference image feature descriptors is determined based on the relationship.
15. The method of claim 10 , further comprising:
ranking one or more sets of reference image feature descriptors, each set being associated with a combination of the determined numbers of duplicate reference image feature descriptors; and
selecting the set of reference image feature descriptors based on a result of the ranking.
16. The method of claim 10 , further comprising:
excluding, from the selection of the one or more reference image feature descriptors, reference image feature descriptors associated with a calculated number of duplicate reference image feature descriptors that is below a predefined threshold.
17. The method of claim 10 , wherein the reference image feature descriptors are stored based on a hash table including a plurality of hash buckets, each hash bucket being associated with an address within the address space; wherein the plurality of functions are configured to map one or more feature descriptors to one or more hash buckets out of the same set of hash buckets.
18. The method of claim 10 , further comprising:
receiving data associated with media material through an interface; and
providing the data as part of the information associated with the selected reference image feature descriptors.
19. A non-transitory computer-readable storage medium having stored thereon program containing instructions that, when executed by a computer, cause the computer to execute a method for determining information relating to images captured by mobile devices, the method comprising:
receiving a first query including a first plurality of query image feature descriptors;
determining, within an address space associated with a plurality of first addresses determined from at least one of the first query image feature descriptors, a number of duplicate reference image feature descriptors,
wherein a plurality of duplicates of at least one reference image feature descriptor is mapped using a plurality of functions to at least a second address;
selecting one or more reference image feature descriptors based on the determined number of duplicate reference image feature descriptors;
determining information associated with the selected reference image feature descriptors;
providing the information associated with the selected reference image feature descriptors to the first mobile device;
receiving a second query from a second mobile device, the second query including the same plurality of query image feature descriptors as in the first query; and
providing the same information associated with the selected reference image feature descriptors to the second mobile device.