Image search device, image search method, program, and computer-readable storage medium
View Patent ↗An image search device includes a common memory and a plurality of parallel processors for executing a same instruction. The image search device transfers, from storage, a plurality of representative feature vectors, which respectively represent a plurality of clusters including a plurality of image feature vectors, stores, in the common memory, one or more query feature vectors extracted from an image serving as a query, calculates a distance between the plurality of transferred representative feature vectors and the query feature vector using the plurality of parallel processors, and selects one or more of a plurality of images based on a distance between the plurality of image feature vectors, which belong to the cluster selected by the calculated distance, and the query feature vector.
1. An image search device, comprising:
a plurality of parallel processors each of which includes a register and which execute a same instruction;
an instruction unit;
a common memory which is commonly and directly accessible from the plurality of parallel processors,
wherein each element of a vector stored in common memory are loaded, at one time, into the register of a corresponding one of the plurality of parallel processors in bulk according to one instruction from the instruction unit, and each of the plurality of parallel processors processes the loaded element of the vector;
a storage means storing a plurality of image feature vectors that are respectively extracted from a plurality of images serving as search targets and respectively belong to one of a plurality of clusters, and a plurality of representative feature vectors each of which represents one of the plurality of clusters, in which more than one of the image feature vectors are extracted from one of the plurality of images;
representative vector transfer means for transferring the plurality of the representative feature vectors from the storage means to the common memory;
query feature vector obtaining means for obtaining and storing, in the common memory, one or more query feature vectors that are extracted from an image serving as a query;
first distance calculating means for calculating a distance between at least a part of the plurality of transferred representative feature vectors and the query feature vector using the plurality of parallel processors;
cluster selecting means selecting a cluster from the plurality of clusters based on the calculation result of the first distance calculating means;
image feature vector transfer means for transferring the image feature vectors, which belongs to the selected cluster from the storage means to the common memory after the cluster unit selects the cluster;
second distance calculating means for calculating a distance between the plurality of image feature vectors, which belong to the cluster selected based on a calculation result of the first distance calculating means, and the query feature vector;
selecting means for selecting at least one of the plurality of images based on a calculation result of the second distance calculating means; and
cluster generator for grouping the plurality of image feature vectors to the plurality of clusters so that a sum of (a number of the plurality of image feature vectors)×(a dimension of the image feature vector)×(a size of one element of the image feature vector)/(the number of the plurality of clusters) and (a number of the plurality of representative feature vectors)×(a dimension of the representative feature vector)×(a size of one element of the representative feature vector) is less than the size of the common memory,
wherein the second distance calculating means calculates a distance between the transferred image feature vectors and the query feature vector using the plurality of parallel processors; and
wherein (a number of the plurality of image feature vectors in the storage means)×(the dimension of the image feature vector)×(the size of one element of the image feature vector) is more than the size of the common memory.
2. The image search device according to claim 1 ,
wherein a data amount of the plurality of representative feature vectors is less than a size of the common memory.
3. The image search device according to claim 2 ,
wherein a data amount of the plurality of image feature vectors that belong to one of the plurality of clusters is less than the size of the common memory, and
wherein a data amount of the plurality of image feature vectors that belong to the plurality of clusters is more than the size of the common memory.
4. The image search device according to claim 1 ,
wherein a data amount of the plurality of image feature vectors that belong to one of the plurality of clusters and the plurality of representative vectors is less than the size of the common memory,
wherein the image feature vector transfer means replaces a plurality of image feature vectors that have previously been stored in the common memory with the plurality of image feature vectors that belong to the selected cluster.
5. The image search device according to claim 1 , further comprising:
image feature vector additional extracting means for extracting a plurality of image feature vectors from an image to be added as a search target; and
image feature vector adding means for adding the plurality of image feature vectors, which are extracted by the image feature vector additional extracting means, to one of the clusters.
6. An image search method for causing a computer, which includes an instruction unit, a plurality of parallel processors each of which includes a register and which executes a same instruction, and a common memory which is commonly and directly accessible from the plurality of parallel processors, wherein each element of a vector stored in the common memory are loaded, at one time, into the register of corresponding one of the plurality of parallel processors in bulk according to one instruction from the instruction unit and each of the plurality of parallel processors processes the loaded element of the vector, to search an image, the method comprising:
a representative vector transfer step for transferring a plurality of representative feature vectors from storage means to the common memory, the storage means storing a plurality of image feature vectors that are respectively extracted from a plurality of images serving as search targets and respectively belong to one of a plurality of clusters, and the plurality of representative feature vectors each of which represents one of the plurality of clusters, in which more than one of the image feature vectors are extracted from one of the images;
a query setting step for setting, in the common memory, one or more query feature vectors extracted from an image serving as a query;
a first distance calculating step for calculating a distance between at least a part of the plurality of transferred representative feature vectors and the query feature vector using the plurality of parallel processors;
a cluster selecting step for selecting a cluster from the plurality of clusters based on the calculation result of the first distance calculating means;
an image feature vector transfer step for transferring the image feature vectors, which belongs to the selected cluster from the storage means to the common memory after the cluster selecting unit selects the cluster;
a second distance calculating step for calculating a distance between the plurality of image feature vectors that belong to the cluster selected based on a calculation result of the first distance calculating step and the query feature vector; and
a selecting step for selecting at least one of the plurality of images based on a calculation result of the second distance calculating step,
a cluster generation step for grouping the plurality of image feature vectors to the plurality of clusters so that a sum of (a number of the plurality of image feature vectors)×(a dimension of the image feature vector)×(a size of one element of the image feature vector)/the number of the plurality of clusters) and (a number of the plurality of representative feature vectors)×(a dimension of the representative feature vector)×(a size of one element of the representative feature vector) is less than the size of the common memory,
wherein the second distance calculating step calculates a distance between the transferred image feature vectors and the query feature vector using the plurality of parallel processors; and
wherein (a number of the plurality of image feature vectors in the storage means)×(the dimension of the image feature vector)×(the size of one element of the image feature vector) is more than the size of the common memory.
7. A non-transitory computer-readable storage medium that stores a program for causing a computer, which includes an instruction unit, a plurality of parallel processors each of which includes a register and which executes a same instruction, and a common memory which is commonly and directly accessible from the plurality of parallel processors wherein each element of a vector stored in the common memory are loaded, at one time, into the register of corresponding one of the plurality of parallel processors in bulk according to one instruction from the instruction unit and each of the plurality of parallel processors processes the loaded element of the vector, to function as:
representative vector transfer means for transferring a plurality of representative feature vectors from storage means to the common memory, the storage means storing a plurality of image feature vectors that are respectively extracted from a plurality of images serving as search targets and respectively belong to one of a plurality of clusters, and the plurality of representative feature vectors each of which represents one of the plurality of clusters, in which more than one of the image feature vectors are extracted from one of the images;
query setting means for setting, in the common memory, one or more query feature vectors extracted from an image serving as a query;
first distance calculating means for calculating a distance between at least a part of the plurality of transferred representative feature vectors and the query feature vector using the plurality of parallel processors;
a cluster selecting means for selecting a cluster from the plurality of clusters based on the calculation result of the first distance calculating means;
an image feature vector transfer means for transferring the image feature vectors, which belongs to the selected cluster from the storage means to the common memory after the cluster selecting unit selects the cluster;
second distance calculating means for calculating a distance between the plurality of image feature vectors that belong to the cluster selected based on a calculation result of the first distance calculating means and the query feature vector; and
selecting means for selecting at least one of the plurality of images based on a calculation result of the second distance calculating means,
cluster generation means for grouping the plurality of image feature vectors to the plurality of clusters so that a sum of (a number of the plurality of image feature vectors)×(a dimension of the image feature vector)×(a size of one element of the image feature vector)/(the number of the plurality of clusters) and (a number of the plurality of representative feature vectors)×(a dimension of the representative feature vector)×(a size of one element of the representative feature vector) is less than the size of the common memory,
wherein the second distance calculating means calculates a distance between the transferred image feature vectors and the query feature vector using the plurality of parallel processors; and
wherein (a number of the plurality of image feature vectors in the storage means)×(the dimension of the image feature vector)×(the size of one element of the image feature vector) is more than the size of the common memory.
8. The image search device according to claim 1 ,
wherein a data amount of the plurality of representative feature vectors is less than a size of the common memory.
9. The image search device according to claim 8 ,
wherein a data amount of the image feature vectors that belong to one of the clusters is less than the size of the common memory, and
wherein a data amount of the image feature vectors that belong to the clusters is more than the size of the common memory.
10. The image search device according to claim 1 , further comprising:
image feature vector additional extracting means for extracting a plurality of image feature vectors from an image to be added as a search target; and
image feature vector adding means for adding the plurality of the image feature vectors, which are extracted by the image feature vector additional extracting means, to one of the clusters.
11. The image search device according to claim 2 , further comprising:
image feature vector additional extracting means for extracting a plurality of image feature vectors from an image to be added as a search target; and
image feature vector adding means for adding the plurality of the image feature vectors, which are extracted by the image feature vector additional extracting means, to one of the clusters.
12. The image search device according to claim 3 , further comprising:
image feature vector additional extracting means for extracting a plurality of image feature vectors from an image to be added as a search target; and
image feature vector adding means for adding the plurality of the image feature vectors, which are extracted by the image feature vector additional extracting means, to one of the clusters.
13. The image search device according to claim 4 , further comprising:
image feature vector additional extracting means for extracting a plurality of image feature vectors from an image to be added as a search target; and
image feature vector adding means for adding the plurality of the image feature vectors, which are extracted by the image feature vector additional extracting means, to one of the clusters.
14. The image search device according to claim 8 , further comprising:
image feature vector additional extracting means for extracting a plurality of image feature vectors from an image to be added as a search target; and
image feature vector adding means for adding the plurality of the image feature vectors, which are extracted by the image feature vector additional extracting means, to one of the clusters.
15. The image search device according to claim 9 , further comprising:
image feature vector additional extracting means for extracting a plurality of image feature vectors from an image to be added as a search target; and
image feature vector adding means for adding the plurality of the image feature vectors, which are extracted by the image feature vector additional extracting means, to one of the clusters.
16. The image search device according to claim 1 ,
wherein the first distance calculating means loads, at one time, each element of one of the plurality of transferred representative feature vectors from the common memory into the register of each of the plurality of parallel processors in which the loaded element is calculated, subtracts, at one time, the element stored in the register from the corresponding elements of one of the one or more query feature vectors and squares the subtraction result, and
wherein the second distance calculating means loads, at one time, each element of one of the plurality of image feature vectors which belong to the selected cluster from the common memory into the register of corresponding one of the plurality of parallel processors in which the loaded element is calculated, subtracts, at one time, the element stored in the register from the corresponding elements of one of the one or more query feature vectors and that squares the subtraction result.
17. The image search device according to claim 1 ,
wherein the representative vector transferring means read the plurality of the representative feature vectors in which the size of each element is 1 byte, and convert the plurality of the representative feature vectors so that the size of each converted element is 4 bytes, and transfer the plurality of the converted representative feature vectors to the common memory.
18. The image search device according to claim 1 ,
wherein the image feature vector transfer means transfers the image feature vectors which belongs to the selected cluster to a predetermined region for one cluster in the common memory if any cluster is selected.
19. The image search device according to claim 1 ,
wherein the image feature vector transfer means transfers the image feature vectors which belong to the selected cluster into a predetermined region for storing one image feature cluster in the common memory each time.
20. The image search device according to claim 1 ,
wherein the image feature vector transfer means transfers the image feature vectors, which belong to the selected cluster from a memory which is not directly accessible from the plurality of parallel processors to the common memory.