Multi-modality based building image retrieval method, device and apparatus

1. A multi-modality based building image retrieval method, comprising:

acquiring a building example image set, a positive prompt word set, and a negative prompt word set inputted by a user,

extracting features from the building example image set by a vision transformer (ViT) image encoder in a contrastive language-image pre-training (CLIP) model, to obtain an image feature set, and calculating a mean image feature of the image feature set,

extracting features from the positive prompt word set and the negative prompt word set by a bidirectional encoder representations from transformers (BERT) text encoder in the CLIP model, to obtain a positive feature set and a negative feature set, and calculating a mean positive feature of the positive feature set and a mean negative feature of the negative feature set,

calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature,

sorting the correlations in descending order, and

selecting top N building images in terms of correlation as target building images, N being a positive integer, wherein

before the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature, the following is further comprised:

extracting image features from the building images in the building image library by the ViT image encoder in the CLIP model, to obtain image feature vectors of the building images; and

extracting text features from the building images in the building image library by the BERT text encoder in the CLIP model, to obtain text feature vectors of the building images; and

a process of the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature comprises:

calculating the correlations between the building example image set and the building images in the building image library according to rel(Q, v)=αI v T I p +βT v T T q −γT v T S b , where Q represents query conditions, v represents building images in a building image library, I v represents image feature vectors of the building images, T v represents text feature vectors of the building images, I v T and T v T represent transposes of I v and T v , respectively, I p represents a mean image feature, T q represents a mean positive feature, S b represents a mean negative feature, and α, β, and γ are manually set hyperparameters, the query conditions Q comprising a mean image feature, a mean positive feature, and a mean negative feature of the building example image set, Q=(I p , T q , Sp).

2. The multi-modality based building image retrieval method according to claim 1 , further comprising:

concatenating the image feature vectors and the text feature vectors corresponding to the building images in the building image library, to obtain an overall representation vector of the building images.

3. The multi-modality based building image retrieval method according to claim 2 , further comprising:

concatenating the mean image feature, the mean positive feature, and the mean negative feature to obtain a query representation vector.

4. The multi-modality based building image retrieval method according to claim 3 , wherein after the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature, the following is further comprised:

performing retrieval based on the overall representation vector of the building images using an approximate nearest neighbor (ANN) indexing algorithm, to obtain an initial building image set.

5. The multi-modality based building image retrieval method according to claim 4 , wherein the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature comprises:

calculating correlations between the building example image set and building images in the initial building image set according to rel(Q, v)=E v T E Q , where Q represents query conditions, v represents building images, E Q represents a query representation vector, E Q =[αI p ; βT q ; −γS b ], I p represents a mean image feature, T q represents a mean positive feature, S b represents a mean negative feature, α, β, and γ are manually set hyperparameters, E v T represents a transpose of E v , and E v represents an overall representation vector of building images.

6. A multi-modality based building image retrieval device, employing a multi-modality based building image retrieval method, the retrieval device comprising:

a condition acquisition unit, configured to acquire a building example image set, a positive prompt word set, and a negative prompt word set inputted by a user;

an image feature extraction unit, configured to extract features from the building example image set by a ViT image encoder in a CLIP model, to obtain an image feature set, and calculate a mean image feature of the image feature set;

a text feature extraction unit, configured to extract features from the positive prompt word set and the negative prompt word set by a BERT text encoder in the CLIP model, to obtain a positive feature set and a negative feature set, and calculate a mean positive feature of the positive feature set and a mean negative feature of the negative feature set;

a correlation calculation unit, configured to calculate correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature;

a sort unit, configured to sort the correlations in descending order; and

an image determination unit, configured to select top N building images in terms of correlation as target building images,

wherein the image feature extraction unit is further configured to extract image features from the building images in the building image library by the ViT image encoder in the CLIP model, to obtain image feature vectors of the building images,

the text feature extraction unit is further configured to extract text features from the building images in the building image library by the BERT text encoder in the CLIP model, to obtain text feature vectors of the building images, and

the correlation calculation unit is further configured to: calculate the correlations between the building example image set and the building images in the building image library according to rel(Q, v)=αI v T I p +βT v T T q −γT v T S b , where Q represents query conditions, v represents building images in a building image library, I v represents image feature vectors of the building images, T v represents text feature vectors of the building images, L v T and T v T represent transposes of L v and T v , respectively, I v represents a mean image feature, T q represents a mean positive feature, S b represents a mean negative feature, and α, β, and γ are manually set hyperparameters, the query conditions Q comprising a mean image feature, a mean positive feature, and a mean negative feature of the building example image set, Q=(L p , T q , S b ).

7. A multi-modality based building image retrieval apparatus,

comprising a memory and a processor;

the memory being configured to store programs; and

the processor being configured to execute the stored programs, and implement a multi-modality based building image retrieval method, comprising:

acquiring a building example image set, a positive prompt word set, and a negative prompt word set inputted by a user,

extracting features from the building example image set by a vision transformer (ViT) image encoder in a contrastive language-image pre-training (CLIP) model, to obtain an image feature set, and calculating a mean image feature of the image feature set,

extracting features from the positive prompt word set and the negative prompt word set by a bidirectional encoder representations from transformers (BERT) text encoder in the CLIP model, to obtain a positive feature set and a negative feature set, and calculating a mean positive feature of the positive feature set and a mean negative feature of the negative feature set,

calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature,

sorting the correlations in descending order, and

selecting top N building images in terms of correlation as target building images, N being a positive integer, wherein

before the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature, the following is further comprised:

extracting image features from the building images in the building image library by the ViT image encoder in the CLIP model, to obtain image feature vectors of the building images; and

extracting text features from the building images in the building image library by the BERT text encoder in the CLIP model, to obtain text feature vectors of the building images; and

a process of the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature comprises:

calculating the correlations between the building example image set and the building images in the building image library according to rel(Q, v)=αI v T I p +βT v T T q −γT v T S b , where Q represents query conditions, v represents building images in a building image library, I v represents image feature vectors of the building images, T v represents text feature vectors of the building images, I v T and T v T represent transposes of I v and T v , respectively, I p represents a mean image feature, T q represents a mean positive feature, S b represents a mean negative feature, and α, β, and γ are manually set hyperparameters, the query conditions Q comprising a mean image feature, a mean positive feature, and a mean negative feature of the building example image set, Q=(L p , T q , S b ).

8. The multi-modality based building image retrieval apparatus according to claim 7 , wherein the multi-modality based building image retrieval method further comprises:

concatenating the image feature vectors and the text feature vectors corresponding to the building images in the building image library, to obtain an overall representation vector of the building images.

9. The multi-modality based building image retrieval apparatus according to claim 8 , wherein the multi-modality based building image retrieval method further comprises:

concatenating the mean image feature, the mean positive feature, and the mean negative feature to obtain a query representation vector.

10. The multi-modality based building image retrieval apparatus according to claim 9 , wherein after the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature, the multi-modality based building image retrieval method further comprises:

performing retrieval based on the overall representation vector of the building images using an approximate nearest neighbor (ANN) indexing algorithm, to obtain an initial building image set.

11. The multi-modality based building image retrieval apparatus according to claim 10 , wherein the calculating correlations between the building example image set and building images in a building image library according to the mean image feature, the mean positive feature, and the mean negative feature comprises:

calculating correlations between the building example image set and building images in the initial building image set according to rel(Q, v)=E v T E Q , where Q represents query conditions, v represents building images, E Q represents a query representation vector, E Q =[αI p ; βT q ; −γS b ], I p represents a mean image feature, T q represents a mean positive feature, S b represents a mean negative feature, α, β, and γ are manually set hyperparameters, E v T represents a transpose of E v , and E v , represents an overall representation vector of building images.