Object detection based on text input that includes both target object classes and target visual attributes
Implementations improve object classification/detection by leveraging visual attributes. An image depicting instance(s) of object class(es) is obtained with a textual snippet that includes: noun(s) identifying target object class(es); and adjective(s) describing target visual attribute(s). The textual snippet may be encoded as text embedding(s) that represent target object class(es) and visual attribute(s) in a shared embedding space. The image may be processed using an image encoder to generate image encoder output tokens (IEOTs) that are used to generate object visual embedding(s) in the shared embedding space. The text embedding(s) and the object visual embeddings may be used to classify the IEOTs as depicting an instance of the target object class(es) having target visual attribute(s). The IEOTs may also be processed using a localization head to predict annotation(s) for the digital image.
1 . A method implemented using one or more processors and comprising:
obtaining a digital image that depicts at least one instance of an agricultural object class of a plurality of agricultural object classes, wherein the at least one instance of the agricultural object class includes a physical attribute of a plurality of physical attributes that is variable amongst objects of the agricultural object class, wherein the plurality of physical attributes includes a ripeness or a health condition of the agricultural object class;
obtaining a textual snippet that includes both:
one or more nouns that identify one or more agricultural object classes of the plurality, and
one or more adjectives that modify the one or more nouns, the one or more adjectives describing one or more physical visual attributes of the plurality for the one or more agricultural object classes, wherein the described physical visual attributes include the ripeness or the health condition;
determining the physical attribute for the instance of the agricultural object class depicted in the digital image, the determination comprising:
generating, using a text encoder, one or more text embeddings in a shared embedding space, each text embedding representing the one or more agricultural object classes and one or more target physical attributes depicted in the digital image;
generating, using an image encoder on the digital image, a plurality of image encoder output tokens;
generating, using an object classification model on the plurality of image encoder output tokens, one or more object visual embeddings in the shared embedding space;
comparing the one or more text embeddings in the shared embedding space with the one or more object visual embeddings in the shared embedding space;
classifying, based on the comparison, one or more of the image encoder output tokens as depicting the instance of the agricultural object class and the physical attribute for the agricultural object class, wherein the physical attribute includes the ripeness or the health condition;
determining, based on the one or more image encoder output tokens classified as depicting the physical attribute for the instance of the agricultural object class in the digital image, an annotation to overlay the instance of the agricultural object class depicted in the digital image; and
applying the at least one determined annotation to the digital image describing the ripeness or the health condition of the identified instance of the agricultural object class in the digital image.
2 . The method of claim 1 , wherein generating the one or more object visual embeddings in the shared embedding space includes linearly projecting the one or more image encoder output tokens into the shared embedding space.
3 . The method of claim 1 , wherein the shared embedding space is a shared contrastive embedding space, and the image and text encoders are pretrained contrastively.
4 . The method of claim 1 , wherein processing the textual snippet based on the text encoder further generates one or more additional text embeddings, each additional text embedding representing a respective one of the one or more target visual attributes.
5 . The method of claim 4 , wherein the comparing comprises performing the following operations separately:
comparing the one or more text embeddings with the one or more object visual embeddings to classify one or more of the image encoder output tokens as depicting an instance of one or more of the target object classes; and
comparing the one or more additional text embeddings with the one or more object visual embeddings to classify one or more of the image encoder output tokens as having one or more of the target visual attributes.
6 . The method of claim 1 , further comprising:
comparing the determined annotation with a manually adjusted annotation of the digital image to determine an error; and
based on the error, training at least the object classification model.
7 . The method of claim 1 , wherein one or both of the text encoder and the image encoder comprises a transformer encoder with a self-attention mechanism.
8 . The method of claim 1 , wherein the determined annotation comprises a bounding box.
9 . The method of claim 1 , wherein the determined annotation comprises a polygon.
10 . A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:
obtain a digital image that depicts at least one instance of an agricultural object class of a plurality of agricultural object classes, wherein the at least one instance of the agricultural object class includes a physical attribute of a plurality of physical attributes that is variable amongst objects of the agricultural object class, wherein the plurality of physical attributes includes a ripeness or a health condition of the agricultural object class;
obtain a textual snippet that includes both:
one or more nouns that identify one or more agricultural object classes of the plurality, and
one or more adjectives that modify the one or more nouns, the one or more adjectives describing one or more physical visual attributes of the plurality for the one or more agricultural object classes, wherein the described physical visual attributes include the ripeness or the health condition;
determine the physical attribute for the instance of the agricultural object class depicted in the digital image, the determination causing the one or more processors to:
generate, using a text encoder, one or more text embeddings in a shared embedding space, each text embedding representing the one or more agricultural object classes and one or more target physical attributes depicted in the digital image;
generate, using an image encoder on the digital image, a plurality of image encoder output tokens;
generate, using an object classification model on the plurality of image encoder output tokens, one or more object visual embeddings in the shared embedding space;
compare the one or more text embeddings in the shared embedding space with the one or more object visual embeddings in the shared embedding space;
classify, based on the comparison, one or more of the image encoder output tokens as depicting the instance of the agricultural object class and the physical attribute for the agricultural object class, wherein the physical attribute includes the ripeness or the health condition;
determine, based on the one or more image encoder output tokens classified as depicting the physical attribute for the instance of the agricultural object class in the digital image, an annotation to overlay the instance of the agricultural object class depicted in the digital image; and
apply the at least one determined annotation to the digital image describing the ripeness or the health condition of the identified instance of the agricultural object class in the digital image.
11 . The system of claim 10 , wherein the instructions to generate the one or more object visual embeddings in the shared embedding space include instructions to linearly project the one or more image encoder output tokens into the shared embedding space.
12 . The system of claim 10 , wherein the shared embedding space is a shared contrastive embedding space, and the image and text encoders are pretrained contrastively.
13 . The system of claim 10 , wherein processing the textual snippet based on the text encoder further generates one or more additional text embeddings, each additional text embedding representing a respective one of the one or more target visual attributes.
14 . The system of claim 13 , wherein the instructions to compare include instructions to separately:
compare the one or more text embeddings with the one or more object visual embeddings to classify one or more of the image encoder output tokens as depicting an instance of one or more of the target object classes; and
compare the one or more additional text embeddings with the one or more object visual embeddings to classify one or more of the image encoder output tokens as having one or more of the target visual attributes.
15 . The system of claim 10 , further comprising instructions to:
compare the determined annotation with a manually adjusted annotation of the digital image to determine an error; and
based on the error, training at least the object classification model.
16 . The system of claim 10 , wherein one or both of the text encoder and the image encoder comprises a transformer encoder with a self-attention mechanism.
17 . The system of claim 10 , wherein the determined annotation comprises a bounding box.
18 . The system of claim 10 , wherein the determined annotation comprises a polygon.
19 . At least one non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to:
obtain a digital image that depicts at least one instance of an agricultural object class of a plurality of agricultural object classes, wherein the at least one instance of the agricultural object class includes a physical attribute of a plurality of physical attributes that is variable amongst objects of the agricultural object class, wherein the plurality of physical attributes includes a ripeness or a health condition of the agricultural object class;
obtain a textual snippet that includes both:
one or more nouns that identify one or more agricultural object classes of the plurality, and
one or more adjectives that modify the one or more nouns, the one or more adjectives describing one or more physical visual attributes of the plurality for the one or more agricultural object classes, wherein the described physical visual attributes include the ripeness or the health condition;
determine the physical attribute for the instance of the agricultural object class depicted in the digital image, the determination causing the one or more processors to:
genearte, using a text encoder, one or more text embeddings in a shared embedding space, each text embedding representing the one or more agricultural object classes and one or more target physical attributes depicted in the digital image;
generate, using an image encoder on the digital image, a plurality of image encoder output tokens;
generate, using an object classification model on the plurality of image encoder output tokens, one or more object visual embeddings in the shared embedding space;
compare the one or more text embeddings in the shared embedding space with the one or more object visual embeddings in the shared embedding space;
classify, based on the comparison, one or more of the image encoder output tokens as depicting the instance of the agricultural object class and the physical attribute for the agricultural object class, wherein the physical attribute includes the ripeness or the health condition;
determine, based on the one or more image encoder output tokens classified as depicting the physical attribute for the instance of the agricultural object class in the digital image, an annotation to overlay the instance of the agricultural object class depicted in the digital image; and
apply the at least one determined annotation to the digital image describing the ripeness or the health condition of the identified instance of the agricultural object class in the digital image.
20 . The at least one non-transitory computer-readable medium of claim 19 , wherein generating the one or more object visual embeddings in the shared embedding space includes linearly projecting the one or more image encoder output tokens into the shared embedding space.