Multi-stage image matching techniques
In some implementations, an image recognition system is capable of using an iterative multi-stage image matching procedure to improve the accuracy and/or reliability of matching a search image with a small number of distinctive feature points to reference images. In the first stage, the system compares a search template for the search image to reference templates of each image within the set of reference images. The system determines a best-matched reference image from among the reference images of the set of reference images. In the second stage, the system compares the hybrid search template to one or more reference templates of reference images that is not the best-matched reference image. The comparison can be used to either compute match scores for reference images corresponding to the one or more reference templates.
1. A method performed by one or more computers, the method comprising:
obtaining a search template comprising (i) search feature points within a search image, and (ii) search descriptors for each of the search feature points;
obtaining a set of reference templates, each reference template within the set of reference templates comprising (i) reference feature points within a particular reference image, and (ii) reference descriptors for each of the reference feature points;
comparing the search template to each reference template within the set of reference templates;
determining a best-matched reference template from among the set of reference templates based on comparing the search template to each reference template within the set of reference templates;
combining one or more reference feature points of the best-matched reference template and one or more search feature points of the search template to generate a hybrid search template;
comparing the hybrid search template to one or more reference templates that is not the best-matched reference template;
computing a respective match score for one or more reference images corresponding to the one or more reference templates that is not the best-matched reference template based on comparing the hybrid search template to the one or more reference templates that is not best-matched reference template; and
providing the match scores for the one or more reference images for output.
2. The method of claim 1 , wherein comparing the search template to each reference template within the set of reference templates comprises:
comparing each search feature point of the search template to each reference feature point of each reference template within the set of reference templates; and
determining, for each reference template within the set of reference templates, a number of mated feature points based on comparing each search feature point of the search template to each reference feature point of each reference template within the set of reference templates, each mated feature point representing a particular reference feature point having one or more reference descriptors that correspond to a search descriptor of a particular search feature point.
3. The method of claim 2 , wherein the best-matched reference template is a particular reference feature template that is determined to have the greatest number of mated feature points from among the reference templates within the set of reference feature templates.
4. The method of claim 1 , wherein the hybrid search template is generated based on combining each reference feature points of the best-matched reference template and each search feature point of the search template.
5. The method of claim 1 , wherein generating the hybrid search template comprises:
comparing each search feature point within the search image to each reference feature point within a reference image corresponding to the best-matched reference template;
determining a mated region of the reference image corresponding to the best-matched reference template based on comparing each search feature point within the search image to each reference feature point within the reference image corresponding to the best-matched reference template, the mated region of the reference image defined by an area of the reference image that includes reference feature points having one or more reference descriptors that correspond to a search descriptor of a particular search feature point;
determining a mated region of the search image based on the mated region of the reference image, the mated region of the search image defined by an area of the search image that includes search feature points having one or more search descriptors that correspond to a reference descriptor of a particular reference feature point; and
combining search feature points included within the mated region of the search image and reference feature points included within the mated region of the reference image to generate the hybrid search template.
6. The method of claim 1 , wherein the one or more reference templates that is not best-matched reference template comprises reference templates included within the set of reference templates that are compared to the search template.
7. The method of claim 1 , wherein the one or more reference templates that is not best-matched reference template comprises reference templates included within a second set of reference templates, the second set of reference templates being associated with a different data source than a data source of the set of reference templates that are compared to the search template.
8. The method of claim 1 , wherein the search feature points or the reference feature points comprises one or more of a ridge intersection point, a ridge edge point, or a corner point.
9. The method of claim 1 , wherein:
the search image comprises an image of an object;
the search feature points comprise search feature points characterizing visual attributes of the object; and
the match scores each represent a likelihood that a reference image corresponding to the one or more reference templates is an image of the object.
10. The method of claim 1 , wherein:
the search image comprises an image of a first object;
the search feature points comprise search feature points characterizing visual attributes of the first object; and
the match scores each represent a likelihood that a reference image corresponding to the one or more reference templates is an image of a second object that is predetermined to be similar to the first object.
11. A system comprising:
one or more computers; and
one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining a search template comprising (i) search feature points within a search image, and (ii) search descriptors for each of the search feature points;
obtaining a set of reference templates, each reference template within the set of reference templates comprising (i) reference feature points within a particular reference image, and (ii) reference descriptors for each of the reference feature points;
comparing the search template to each reference template within the set of reference templates;
determining a best-matched reference template from among the set of reference templates based on comparing the search template to each reference template within the set of reference templates;
combining one or more reference feature points of the best-matched reference template and one or more search feature points of the search template to generate a hybrid search template;
comparing the hybrid search template to one or more reference templates that is not the best-matched reference template;
computing a respective match score for one or more reference images corresponding to the one or more reference templates that is not the best-matched reference template based on comparing the hybrid search template to the one or more reference templates that is not best-matched reference template; and
providing the match scores for the one or more reference images for output.
12. The system of claim 11 , wherein comparing the search template to each reference template within the set of reference templates comprises:
comparing each search feature point of the search template to each reference feature point of each reference template within the set of reference templates; and
determining, for each reference template within the set of reference templates, a number of mated feature points based on comparing each search feature point of the search template to each reference feature point of each reference template within the set of reference templates, each mated feature point representing a particular reference feature point having one or more reference descriptors that correspond to a search descriptor of a particular search feature point.
13. The system of claim 12 , wherein the best-matched reference template is a particular reference feature template that is determined to have the greatest number of mated feature points from among the reference templates within the set of reference feature templates.
14. The system of claim 11 , wherein the hybrid search template is generated based on combining each reference feature points of the best-matched reference template and each search feature point of the search template.
15. The system of claim 11 , wherein generating the hybrid search template comprises:
comparing each search feature point within the search image to each reference feature point within a reference image corresponding to the best-matched reference template;
determining a mated region of the reference image corresponding to the best-matched reference template based on comparing each search feature point within the search image to each reference feature point within the reference image corresponding to the best-matched reference template, the mated region of the reference image defined by an area of the reference image that includes reference feature points having one or more reference descriptors that correspond to a search descriptor of a particular search feature point;
determining a mated region of the search image based on the mated region of the reference image, the mated region of the search image defined by an area of the search image that includes search feature points having one or more search descriptors that correspond to a reference descriptor of a particular reference feature point; and
combining search feature points included within the mated region of the search image and reference feature points included within the mated region of the reference image to generate the hybrid search template.
16. A non-transitory computer-readable storage device encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
obtaining a search template comprising (i) search feature points within a search image, and (ii) search descriptors for each of the search feature points;
obtaining a set of reference templates, each reference template within the set of reference templates comprising (i) reference feature points within a particular reference image, and (ii) reference descriptors for each of the reference feature points;
comparing the search template to each reference template within the set of reference templates;
determining a best-matched reference template from among the set of reference templates based on comparing the search template to each reference template within the set of reference templates;
combining one or more reference feature points of the best-matched reference template and one or more search feature points of the search template to generate a hybrid search template;
comparing the hybrid search template to one or more reference templates that is not the best-matched reference template;
computing a respective match score for one or more reference images corresponding to the one or more reference templates that is not the best-matched reference template based on comparing the hybrid search template to the one or more reference templates that is not best-matched reference template; and
providing the match scores for the one or more reference images for output.
17. The device of claim 16 , wherein comparing the search template to each reference template within the set of reference templates comprises:
comparing each search feature point of the search template to each reference feature point of each reference template within the set of reference templates; and
determining, for each reference template within the set of reference templates, a number of mated feature points based on comparing each search feature point of the search template to each reference feature point of each reference template within the set of reference templates, each mated feature point representing a particular reference feature point having one or more reference descriptors that correspond to a search descriptor of a particular search feature point.
18. The device of claim 17 , wherein the best-matched reference template is a particular reference feature template that is determined to have the greatest number of mated feature points from among the reference templates within the set of reference feature templates.
19. The device of claim 16 , wherein the hybrid search template is generated based on combining each reference feature points of the best-matched reference template and each search feature point of the search template.
20. The device of claim 16 , wherein generating the hybrid search template comprises:
comparing each search feature point within the search image to each reference feature point within a reference image corresponding to the best-matched reference template;
determining a mated region of the reference image corresponding to the best-matched reference template based on comparing each search feature point within the search image to each reference feature point within the reference image corresponding to the best-matched reference template, the mated region of the reference image defined by an area of the reference image that includes reference feature points having one or more reference descriptors that correspond to a search descriptor of a particular search feature point;
determining a mated region of the search image based on the mated region of the reference image, the mated region of the search image defined by an area of the search image that includes search feature points having one or more search descriptors that correspond to a reference descriptor of a particular reference feature point; and
combining search feature points included within the mated region of the search image and reference feature points included within the mated region of the reference image to generate the hybrid search template.