Method and system to perform text-to-image queries with wildcards
View Patent ↗A system and method for comparing a text image with or without a wildcard character and a character string are provided. The method includes embedding a character string into a vectorial space by extracting a set of features from the character string and generating a character string representation based on the extracted features, such as a spatial pyramid bag of characters (SPBOC) representation. A text image is embedded into a vectorial space by extracting a set of features from the text image and generating a text image representation based on the text image extracted features. A similarity between the text image representation and the character string representation is computed, which includes computing a function of the text image representation and character string representation.
1. A system for comparing a queried character string including a wildcard character and a text image comprising:
a character string image representation generator configured to embed a character string image representation of the queried character string including the wildcard into a common vectorial space based on features extracted from the queried character string including the wildcard character, the queried character string consisting of a sequence of a plurality of characters including the wildcard character, and the character string image representation including a spatial pyramid bag of characters representation including information indicating a location of the wildcard character relative to other characters included in the queried character string;
a text image representation generator configured to embed a text image representation of the text image into the common vectorial space, the text image based on features extracted from the text image, and the text image including a sequence of a plurality of textual characters;
a comparator configured to compute a similarity between the text image representation embedded in the common vectorial space and the character string image representation embedded in the common vectorial space;
an output component configured to output information based on the computed similarity between the character string image representation embedded in the common vectorial space and the text image representation embedded in the common vectorial space; and
a processor which implements the text string representation generator, text image representation generator, comparator, and output component.
2. A computer implemented method for comparing a queried character string and a text image comprising:
for at least one queried character string including a sequence of characters including a wildcard character, extracting a set of features from the queried character string by partitioning the queried character string to form a spatial pyramid of regions and for each region generating a representation of the region including counting occurrences of each of a finite set of characters in the region;
generating a character string image representation of the queried character string based on the region representations, the character string image representation including information indicating a location of the wildcard character relative to other characters included in the queried character string;
embedding the character string image representation into a common vectorial space;
extracting a set of features from the text image and generating a text image representation based thereon;
embedding the text image into the common vectorial space; and
computing a similarity function between the text image representation and the character string representation comprising embedding at least one of the character string representation and the text image representation with a matrix of learned parameters, the similarity being a function of the at least one embedded representation, and
the learned parameters of the similarity function learned using a training set of text images that are each labeled with a respective character string including a wildcard character corresponding to text present in the image.
3. The method of claim 1 , further comprising learning parameters of the similarity function using a training set of text images that are each labeled with a respective character string including a wildcard character corresponding to text present in the image.