Method, apparatus, electronic device, and storage medium for image-based data processing

1. A method for image-based data processing, the method comprising:

inputting a given image and a query, the given image comprising a plurality of objects comprising a first object and a second object determining attributes associated with the objects presented in the given image from a predefined data set, the data set comprising pre-annotated objects and attributes of the pre-annotated objects, wherein determining the attributes associated with the objects presented in the image comprises: determining the first object and the second object presented in the given image by recognizing the given image, and determining a first attribute of the first object and the second attribute of the second object;

determining degrees of correlation between the objects and the query based on the objects and the attribute, wherein determining the degrees of correlation between the objects and the queries comprises: determining a first degree of correlation between the first object and the query by comparing the first object and the first attribute with the query, and determining a second degree of correlation between the second object and the query by comparing the second object and the second attribute with the query;

ranking the first object and the second object by comparing the first degree and the second degree; and

searching for and providing a response to the query based on the degrees of correlation, wherein the response has a higher relevance to a higher ranked object in the first object and the second object, wherein the method is performed by a processor.

2. The method according to claim 1 , wherein the determining the degrees of correlation between the objects and the query comprises:

determining a first feature representation of a combination of the object and the attribute;

determining at least one second feature representation corresponding to at least one description unit in the query; and

obtaining the degree of correlation based on the first feature representation and the at least one second feature representation.

3. The method according to claim 2 , wherein the at least one second feature representation comprises a plurality of second feature representations, and the obtaining the degree of correlation comprises:

computing a plurality of similarities between the first feature representation and the plurality of second feature representations respectively; and

obtaining the degree of correlation based on the plurality of similarities.

4. The method according to claim 3 , wherein the obtaining the degree of correlation comprises:

selecting at least one of a maximum value, a median, an average value, or a weighted value of the plurality of similarities for use as the degree of correlation.

5. The method according to claim 1 , wherein the providing a response to the query comprises:

determining a first weight of the first object and a second weight of the second object based on the first degree and the second degree; and

determining the response based on the first weight of the first object and the second weight of the second object.

6. An apparatus for image-based data processing, comprising:

at least one processor; and

a memory storing instructions, wherein the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:

inputting a given image and a query, the given image comprising a plurality of objects comprising a first object and a second object

determining attributes associated with the objects presented in the given image from a predefined data set, the data set comprising pre-annotated objects and attributes of the pre-annotated object, wherein determining the attributes associated with the objects presented in the image comprises: determining the first object and the second object presented in the given image by recognizing the given image, and determining a first attribute of the first object and the second attribute of the second object;

determining degrees of correlation between the objects and the query based on the objects and the attribute, wherein determining the degrees of correlation between the objects and the queries comprises: determining a first degree of correlation between the first object and the query by comparing the first object and the first attribute with the query, and determining a second degree of correlation between the second object and the query by comparing the second object and the second attribute with the query;

ranking the first object and the second object by comparing the first degree and the second degree; and

searching for and providing a response to the query based on the degrees of correlation, wherein the response has a higher relevance to a higher ranked object in the first object and the second object.

7. The apparatus according to claim 6 , wherein the determining the degrees of correlation between the objects and the query comprises:

determining a first feature representation of a combination of the object and the attribute;

determining at least one second feature representation corresponding to at least one description unit in the query; and

obtaining the degree of correlation based on the first feature representation and the at least one second feature representation.

8. The apparatus according to claim 7 , wherein the at least one second feature representation comprises a plurality of second feature representations, and the correlation degree obtaining module comprises:

computing a plurality of similarities between the first feature representation and the plurality of second feature representations respectively; and

obtaining the degree of correlation based on the plurality of similarities.

9. The apparatus according to claim 8 , wherein the obtaining the degree of correlation comprises:

selecting at least one of a maximum value, a median, an average value, or a weighted value of the plurality of similarities for use as the degree of correlation.

10. The apparatus according to claim 6 , wherein the providing a response to the query comprises:

determining a first weight of the first object and a second weight of the second object based on the first degree and the second degree; and

determining the response based on the first weight of the first object and the second weight of the second object.

11. A non-transitory computer readable storage medium, storing a computer program thereon, wherein the program, when executed by a processor, causes the processor to perform operations, the operations comprising:

inputting a given image and a query, the given image comprising a plurality of objects comprising a first object and a second object;

determining attributes associated with the objects presented in the given image from a predefined data set, the data set comprising pre-annotated objects and attributes of the pre-annotated object, wherein determining the attributes associated with the objects presented in the image comprises: determining the first object and the second object presented in the given image by recognizing the given image, and determining a first attribute of the first object and the second attribute of the second object;

determining degrees of correlation between the objects and the query based on the objects and the attribute, wherein determining the degrees of correlation between the objects and the queries comprises: determining a first degree of correlation between the first object and the query by comparing the first object and the first attribute with the query, and determining a second degree of correlation between the second object and the query by comparing the second object and the second attribute with the query;

ranking the first object and the second object by comparing the first degree and the second degree; and

searching for and providing a response to the query based on the degree of correlation, wherein the response has a higher relevance to a higher ranked object in the first object and the second object.

12. The method according to claim 1 , wherein the response is an answer using a natural language.

13. The method according to claim 1 , wherein the method is applied to a visual question answering system, the given image and the query are inputs of the system, and the response is an output of the system responding to the inputs and is an answer using a natural language.