Interaction detection model for identifying human-object interactions in image content

1. A method that includes one or more processing devices performing operations comprising:

accessing, from a memory device, an input image;

transforming the input image by applying human-object interaction metadata to the input image that identifies a part of a human depicted in the input image being in contact with a part of an object depicted in the input image, wherein applying the human-object interaction metadata comprises:

providing the input image to an interaction detection network having a pose estimation subnet, an object contact subnet, and an interaction-detection subnet,

computing a joint-location heat map for the input image using the pose estimation subnet, the joint-location heat map identifying one or more human joint locations in the input image,

computing a contact-point heat map for the input image using the object contact subnet, the contact-point heat map identifying one or more contact points on the object depicted in the input image,

combining the joint-location heat map and the contact-point heat map into a combined heat map image, wherein the combined heat map image includes a joint-object contact pair, and wherein the joint-object contact pair comprises a relationship between a human joint location and a contact point, and

generating the human-object interaction metadata for the combined heat map image using the interaction-detection subnet, wherein the interaction-detection subnet is trained to identify an interaction based on the joint-object contact pair; and

providing access to the input image having the human-object interaction metadata.

2. The method of claim 1 , the operations further comprising training the interaction-detection subnet, wherein training the interaction-detection subnet comprises:

receiving, by the interaction-detection subnet, a plurality of training inputs, each training input comprising a training image and training human-object interaction metadata;

computing, by the interaction-detection subnet, a calculated human-object interaction metadata from the plurality of training inputs;

detecting, by a training computing system, a difference between the calculated human-object interaction metadata and the training human-object interaction metadata; and

updating, by the training computing system, one or more parameters used to generate human-object interaction metadata, wherein updating the one or more parameters decreases a subsequent difference between the training human-object interaction metadata and subsequent human-interaction metadata calculated with the updated one or more parameters.

3. The method of claim 2 , the operations further comprising:

receiving, through an input device, feedback from a user device indicating whether the human-object interaction metadata is accurate; and

updating, by the training computing system, one or more parameters used to generate human-object interaction metadata based on the feedback.

4. The method of claim 1 , the operations further comprising servicing an image query using an image search system, wherein servicing the image query comprises:

receiving, through an input device, the image query comprising one or more search terms;

determining, by the image search system, a match score by comparing the human-object interaction metadata to the one or more search terms from the image query; and

transmitting the input image to the image search system based on the match score exceeding a threshold.

5. The method of claim 1 , wherein generating the human-object interaction metadata further comprises determining that the joint-object contact pair identifies a location of the human joint location and a location of the contact point to be sufficiently similar.

6. The method of claim 5 , wherein determining that the joint-object contact pair to be sufficiently similar comprises determining that a dot product of the human joint location and the location of the contact point exceeds a threshold dot product.

7. The method of claim 1 , wherein the transforming the input image further comprises:

providing the input image to an image evaluation network; and

determining, by the image evaluation network, an estimated human-object interaction, wherein generating the human-object interaction metadata further comprises combining the estimated human-object interaction with the identification of the interaction by the interaction-detection subnet.

8. A system comprising:

a processing device; and

a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program code stored in the non-transitory computer-readable medium and thereby perform operations comprising:

transforming an input image by applying human-object interaction metadata to the input image that identifies a part of a human depicted in the input image being in contact with a part of an object depicted in the input image, wherein applying the human-object interaction metadata comprises:

providing the input image to an interaction detection network having a pose estimation subnet, an object contact subnet, and an interaction-detection subnet,

computing a joint-location heat map for the input image using the pose estimation subnet, the joint-location heat map identifying one or more human joint locations in the input image,

computing a contact-point heat map for the input image using the object contact subnet, the contact-point heat map identifying one or more contact points on the object depicted in the input image, wherein the one or more contact points represent a subset of the object that the human can interact with,

generating the human-object interaction metadata by:

(i) applying the interaction-detection subnet to the joint-location heat map and the contact-point heat map, wherein the interaction-detection subnet is trained to identify an interaction based on joint-object contact pairs, and wherein a joint-object contact pair comprises a relationship between a human joint location and a contact point, and

(ii) determining that the joint-object contact pair identifies a location of the human joint location and a location of the contact point to be sufficiently similar; and

providing a computing system with access to the input image having the human-object interaction metadata.

9. The system of claim 8 , further comprising a training computing system configured for training the interaction-detection subnet, wherein training the interaction-detection subnet comprises:

receiving, by the interaction-detection subnet, a plurality of training inputs, each training input comprising a training image and training human-object interaction metadata;

computing, by the interaction-detection subnet, a calculated human-object interaction metadata from the plurality of training inputs;

detecting a difference between the calculated human-object interaction metadata and the training human-object interaction metadata; and

updating one or more parameters used to generate human-object interaction metadata, wherein updating the one or more parameters decreases a subsequent difference between the training human-object interaction metadata and subsequent human-interaction metadata calculated with the updated one or more parameters.

10. The system of claim 9 , the operations further comprising:

receiving, via an input device, feedback from a user device indicating whether the human-object interaction metadata is accurate; and

wherein the training computing system is further configured for updating one or more parameters used to generate human-object interaction metadata based on the feedback.

11. The system of claim 8 , further comprising an image search system configured for servicing an image query, wherein servicing the image query comprises:

receiving the image query comprising one or more search terms;

determining a match score by comparing the human-object interaction metadata to the one or more search terms from the image query; and

transmitting the input image to the image search system based on the match score exceeding a threshold.

12. The system of claim 8 , wherein applying the human-object interaction metadata further comprises combining the joint-location heat map and the contact-point heat map into a combined heat map image.

13. The system of claim 8 , wherein determining that the joint-object contact pair to be sufficiently similar comprises determining that a dot product of the human joint location and the location of the contact point exceeds a threshold dot product.

14. The system of claim 8 , wherein the transforming the input image further comprises:

providing the input image to an image evaluation network; and

determining, by the image evaluation network, an estimated human-object interaction, wherein generating the human-object interaction metadata further comprises combining the estimated human-object interaction with the identification of the interaction by the interaction-detection subnet.

15. A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising:

a step for accessing an input image from a memory device;

transforming the input image by applying human-object interaction metadata to the input image that identifies a part of a human depicted in the input image being in contact with a part of an object depicted in the input image, wherein applying the human-object interaction metadata comprises:

providing the input image to an interaction detection network having a pose estimation subnet, an object contact subnet, and an interaction-detection subnet,

a step for computing a joint-location heat map for the input image using the pose estimation subnet, the joint-location heat map identifying one or more human joint locations in the input image,

a step for computing a contact-point heat map for the input image using the object contact subnet, the contact-point heat map identifying one or more contact points on the object depicted in the input image, wherein the one or more contact points correspond to a subset of an object that a person can interact with, and

a step for generating the human-object interaction metadata by applying the interaction-detection subnet to the joint-location heat map and the contact-point heat map, wherein the interaction-detection subnet is trained to identify an interaction based on joint-object contact pairs, and wherein a joint-object contact pair comprises a location of a human joint location and a location of the contact point that includes a relationship between the human joint location and a contact point that is sufficiently similar; and

a step for providing an image search system with access to the input image having the human-object interaction metadata.

16. The non-transitory computer-readable medium of claim 15 , the operations further comprising training the interaction-detection subnet, wherein training the interaction-detection subnet comprises:

receiving, by the interaction-detection subnet, a plurality of training inputs, each training input comprising a training image and training human-object interaction metadata;

computing, by the interaction-detection subnet, a calculated human-object interaction metadata from the plurality of training inputs;

detecting a difference between the calculated human-object interaction metadata and the training human-object interaction metadata; and

updating one or more parameters used to generate human-object interaction metadata, wherein updating the one or more parameters decreases a subsequent difference between the training human-object interaction metadata and subsequent human-interaction metadata calculated with the updated one or more parameters.

17. The non-transitory computer-readable medium of claim 16 , the operations further comprising:

receiving, through an input device, feedback from a user device indicating whether the human-object interaction metadata is accurate; and

updating, by the interaction-detection subnet, one or more parameters used to generate human-object interaction metadata based on the feedback.

18. The non-transitory computer-readable medium of claim 15 , wherein generating the human-object interaction metadata further comprises determining that the joint-object contact pair identifies a location of the human joint location and a location of the contact point to be sufficiently similar.

19. The non-transitory computer-readable medium of claim 18 , wherein determining that the joint-object contact pair to be sufficiently similar comprises determining that a dot product of the human joint location and the location of the contact point exceeds a threshold dot product.

20. The non-transitory computer-readable medium of claim 15 , wherein the transforming the input image further comprises:

providing the input image to an image evaluation network; and

determining, by the image evaluation network, an estimated human-object interaction, wherein generating the human-object interaction metadata further comprises combining the estimated human-object interaction with the identification of the interaction by the interaction-detection subnet.