Image annotation

1. A system, comprising:

a processor; and

storage storing computer-readable instructions which, when executed by the processor, cause the processor to:

receive identification of a query image,

generate retrieval-based image annotations for the query image by:

initiating a search using the query image to identify images other than the query image,

obtaining textual metadata for the identified images,

constructing a retrieval-based vocabulary from the textual metadata,

determining visualness scores for various terms, the visualness scores reflecting extents to which individual terms relate to visual image content;

removing low visualness terms having corresponding visualness scores below a threshold from the retrieval-based vocabulary, and

after removing the low visualness terms from the retrieval-based vocabulary, pairing terms from the retrieval-based vocabulary with the identified images as the retrieval-based image annotations,

generate model-based image annotations for the query image by:

inputting the query image to a trained model that has been trained using training images having labels selected from a model-based vocabulary that lacks at least some terms of the retrieval-based vocabulary, and

obtaining the model-based image annotations from the trained model, wherein the trained model predicts that the model-based image annotations apply to the query image and the model-based image annotations are selected from the model-based vocabulary,

integrate and rank the retrieval-based image annotations and the model-based image annotations to generate a diversified hybrid image annotation result for the query image, the diversified hybrid image annotation result comprising at least one of the retrieval-based image annotations and at least one of the model-based image annotations ranked relative to one another; and

output the diversified hybrid image annotation result in response to the query image.

2. The system of claim 1 , wherein the computer-readable instructions further cause the processor to:

initiate a search for the identified images by sending the query image to a search engine that retrieves the identified images based at least on the query image,

wherein the search engine re-indexes as new images are provided to the search engine.

3. The system of claim 1 , wherein the computer-readable instructions further cause the processor to:

select the retrieval-based image annotations based on scoring from a diversified annotation scoring metric.

4. The system of claim 3 , wherein the diversified annotation scoring metric comprises a discounted cumulative gain calculation.

5. The system of claim 1 , wherein the retrieval-based image annotations and the model-based image annotations are integrated and ranked by applying a hybridized annotation scoring metric.

6. The system of claim 5 , wherein the retrieval-based image annotations and the model-based image annotations are ranked based at least in part on analysis of feature gain over multiple features.

7. The system of claim 5 , wherein the computer-readable instructions further cause the processor to:

generate a presentation of the diversified hybrid image annotation result.

8. The system of claim 7 , wherein the presentation of the diversified hybrid image annotation result includes the query image, at least one of the identified images, the at least one of the retrieval-based image annotations, and the at least one of the model-based image annotations.

9. The system of claim 8 , wherein the computer-readable instructions further cause the processor to:

order the at least one of the retrieval-based image annotations and the at least one of the model-based image annotations on the presentation based at least in part on rankings from the hybridized annotation scoring metric applied to the retrieval-based image annotations and the model-based image annotations.

10. The system of claim 1 , wherein the trained model comprises a deep neural network.

11. The system of claim 10 , wherein the computer-readable instructions further cause the processor to:

train the deep neural network using the training images and the labels selected from the model-based vocabulary.

12. The system of claim 1 , wherein the computer-readable instructions further cause the processor to:

present the output to a user.

13. The system of claim 1 , wherein the computer-readable instructions further cause the processor to:

generate the model-based image annotations for the query image using multiple trained models.

14. The system of claim 13 , wherein the computer-readable instructions further cause the processor to:

automatically trigger use of at least one custom trained model to generate the model-based image annotations based at least on visual content of the query image.

15. A method comprising:

receiving identification of a query image,

generating retrieval-based image annotations for the query image by:

initiating a search using the query image to identify images other than the query image,

obtaining textual metadata for the identified images,

constructing a retrieval-based vocabulary from the textual metadata,

determining visualness scores for various terms, the visualness scores reflecting extents to which individual terms relate to visual image content;

removing low visualness terms having corresponding visualness scores below a threshold from the retrieval-based vocabulary, and

after removing the low visualness terms from the retrieval-based vocabulary, pairing terms from the retrieval-based vocabulary with the identified images as the retrieval-based image annotations,

generating model-based image annotations for the query image by:

inputting the query image to a trained model that has been trained using training images having labels selected from a model-based vocabulary that lacks at least some terms of the retrieval-based vocabulary, and

obtaining the model-based image annotations from the trained model, wherein the trained model predicts that the model-based image annotations apply to the query image and the model-based image annotations are selected from the model-based vocabulary,

integrating and ranking the retrieval-based image annotations and the model-based image annotations to generate a diversified hybrid image annotation result for the query image, the diversified hybrid image annotation result comprising at least one of the retrieval-based image annotations and at least one of the model-based image annotations ranked relative to one another; and

outputting the diversified hybrid image annotation result in response to the query image.

16. The method of claim 15 , further comprising:

selecting the retrieval-based image annotations based on scoring from a diversified annotation scoring metric.

17. The method of claim 16 , wherein the diversified annotation scoring metric comprises a discounted cumulative gain calculation.

18. The method of claim 15 , wherein the retrieval-based image annotations and the model-based image annotations are integrated and ranked by applying a hybridized annotation scoring metric.

19. The method of claim 15 , wherein the trained model comprises a deep neural network.

20. A computer-readable storage medium storing computer-readable instructions which, when executed by a processor, cause the processor to perform acts comprising:

receiving identification of a query image,

generating retrieval-based image annotations for the query image by:

initiating a search using the query image to identify images other than the query image,

obtaining textual metadata for the identified images,

constructing a retrieval-based vocabulary from the textual metadata,

determining visualness scores for various terms, the visualness scores reflecting extents to which individual terms relate to visual image content;

removing low visualness terms having corresponding visualness scores below a threshold from the retrieval-based vocabulary, and

after removing the low visualness terms from the retrieval-based vocabulary, pairing terms from the retrieval-based vocabulary with the identified images as the retrieval-based image annotations,

generating model-based image annotations for the query image by:

inputting the query image to a trained model that has been trained using training images having labels selected from a model-based vocabulary that lacks at least some terms of the retrieval-based vocabulary, and

obtaining the model-based image annotations from the trained model, wherein the trained model predicts that the model-based image annotations apply to the query image and the model-based image annotations are selected from the model-based vocabulary,

integrating and ranking the retrieval-based image annotations and the model-based image annotations to generate a diversified hybrid image annotation result for the query image, the diversified hybrid image annotation result comprising at least one of the retrieval-based image annotations and at least one of the model-based image annotations ranked relative to one another; and

outputting the diversified hybrid image annotation result in response to the query image.