Decompositional learning for color attribute prediction
The present disclosure describes a model for large scale color prediction of objects identified in images. Embodiments of the present disclosure include an object detection network, an attention network, and a color classification network. The object detection network generates object features for an object in an image and may include a convolutional neural network (CNN), region proposal network, or a ResNet. The attention network generates an attention vector for the object based on the object features, wherein the attention network takes a query vector based on the object features, and a plurality of key vector and a plurality of value vectors corresponding to a plurality of colors as input. The color classification network generates a color attribute vector based on the attention vector, wherein the color attribute vector indicates a probability of the object including each of the plurality of colors.
1. A method comprising:
receiving a query with a color term corresponding to a reference color;
computing object features for an object in an image;
computing a query vector based on the object features;
obtaining a key vector that transforms the reference color to color features in a feature space of the object features, wherein the key vector is obtained independent of the image;
generating a color attribute vector using a machine learning model that performs an attention mechanism based on the key vector and the query vector, wherein the color attribute vector indicates a probability of the object including the reference color; and
transmitting a response indicating that the image matches the query based on the color attribute vector.
2. The method of claim 1 , further comprising:
computing image features for the image; and
identifying a bounding box for the object, wherein the object features are computed based on the image features and the bounding box.
3. The method of claim 2 , further comprising:
performing region-of-interest pooling on the image features based on the bounding box, wherein the object features are based on the region-of-interest pooling.
4. The method of claim 1 , further comprising:
identifying a single-color image for each of a plurality of colors; and
generating the color features based on each of a plurality of single-color images, wherein the key vector and a value vector are based on the color features.
5. The method of claim 4 , wherein:
one of the plurality of colors corresponds to a human-perceived color of the object in context of the image.
6. The method of claim 1 , further comprising:
identifying a plurality of objects having an object attribute corresponding to a color;
generating object features for each of the plurality of objects; and
averaging the object features to produce the color features, wherein the key vector and a value vector are based on the color features.
7. The method of claim 1 , further comprising:
calculating color features based on a running mean with a momentum factor, wherein the key vector and a value vector are generated based at least in part on the color features.
8. The method of claim 1 , wherein:
the key vector and a value vector are computed independent of the image.
9. The method of claim 1 , further comprising:
indexing images in a database based on the color attribute vector;
receiving a search query including a color attribute; and
retrieving image search results from the indexed images based on the color attribute.