Semantic visual search engine
View Patent ↗An improved method, device and computer program product for enabling a system to learn, categorize and search items such as images and video clips according to their semantic meanings. According to the present invention, prominent features can be separated from low-level features in an item using supervised learning approaches. Prominent features are used to categorize and annotate new target items. Users can then use key words and/or template items for searching through the respective database.
1. A method of categorizing a plurality of items on a mobile electronic device, comprising:
converting, by using a computer, a plurality of items into a plurality of candidate low-level features, for each of the plurality of items, the candidate low-level features being extracted locally around salient points in respective item;
using a supervised learning approach to select prominent low-level features from the plurality of candidate low-level features, the prominent low-level features being associated with predefined object categories;
converting a target item into a plurality of multi-scale local features; and
for each of the plurality of multi-scale local features, matching the multi-scale local features with the prominent low-level features using a probabilistic model.
2. The method of claim 1 , wherein the supervised learning approach comprises the Adaptive Boosting (AdaBoosting) learning algorithm.
3. The method of claim 1 , wherein the supervised learning approach comprises the use of Bayesian statistics.
4. The method of claim 1 , wherein the supervised learning approach comprises support vector machine (SVM) pattern recognition.
5. The method of claim 1 , wherein the plurality of items comprise videos.
6. The method of claim 1 , wherein the plurality of items comprise images.
7. The method of claim 1 , wherein a multi-scale local feature is matched with a prominent low-level feature if a computed probability of a match between the multi-scale local feature and the prominent low-level feature is greater than a predetermined threshold.
8. The method of claim 7 , wherein the predetermined threshold is determined through the use of the supervised learning approach.
9. The method of claim 1 , further comprising: receiving an input from a user; and returning at least one item to the user that shares certain similarities with the input.
10. The method of claim 9 , wherein the input comprises a keyword, and wherein the at least one item includes an annotation that is similar to the keyword.
11. The method of claim 9 , wherein the input comprises a template item, and wherein low-level features in the input are compared to the prominent low-level features to identify the at least one item.
12. The method of claim 9 , wherein the input comprises a template item and the keyword, and the returning of the at least one item comprises:
pre-filtering low-level features in the template item using probabilistic models of interested categories; and
matching the pre-filtered low-level features with target images in the same category, the category being identified by the keyword.
13. A computer program product, embodied on a memory, for categorizing a plurality of items on a mobile electronic device, the computer program product comprising:
computer code for converting a plurality of items into a plurality of candidate low-level features, for each of the plurality of items, the candidate low-level features being extracted locally around salient points in respective item;
computer code for using a supervised learning approach to select prominent low-level features from the plurality of candidate low-level features, the prominent low-level features being associated with predefined object categories;
computer code for converting a target item into a plurality of multi-scale local features and
computer code for, for each of the plurality of multi-scale local features, matching the multi-scale local features with the prominent low-level features using a probabilistic model.
14. The computer program product of claim 13 , wherein a multi-scale local feature is matched with a prominent low-level feature if a computed probability of a match between the multi-scale local feature and the prominent low-level feature is greater than a predetermined threshold.
15. The computer program product of claim 14 , wherein the predetermined threshold is determined through the use of the supervised learning approach.
16. The computer program product of claim 13 , further comprising: computer code for receiving an input from a user; and computer code for returning at least one item to the user that share certain similarities with the input.
17. The computer program product of claim 16 , wherein the input comprises a keyword, and wherein the at least one item includes an annotation that is similar to the keyword.
18. The computer program product of claim 16 , wherein the input comprises a template item, and wherein low-level features in the input are compared to the prominent low-level features to identify the at least one item.
19. The computer program product of claim 16 , wherein the input comprises a template item and the keyword, and the returning of the at least one item comprises:
computer code for pre-filtering low-level features in the template item using probabilistic models of interested categories; and
computer code for matching the pre-filtered low-level features with target images in the same category, the category being identified by the keyword.
20. An electronic device, comprising:
a device unit and
a memory operatively connected to the device unit and the memory including:
computer code for converting a plurality of items into a plurality of candidate low-level features, for each of the plurality of items, the candidate low-level features being extracted locally around salient points in respective item;
computer code for using a supervised learning approach to select prominent low-level features from the plurality of candidate low-level features, the prominent low-level features being associated with predefined object categories;
computer code for converting a target item into a plurality of multi-scale local features; and
computer code for, for each of the plurality of multi-scale local features, matching the multi-scale local features with the prominent low-level features using a probabilistic model.
21. The electronic device of claim 20 , wherein a multi-scale local feature is matched with a prominent low-level feature if a computed probability of a match between the multi-scale local feature and the prominent low-level feature is greater than a predetermined threshold.
22. The electronic device of claim 21 , wherein the predetermined threshold is determined through the use of the supervised learning approach.
23. The electronic device of claim 22 , wherein the memory unit further comprises:
computer code for receiving an input from a user; and
computer code for returning at least one item to the user that shares certain similarities with the input.