Context-driven learning of human-object interactions
A method for classifying a human-object interaction includes identifying a human-object interaction in the input. Context features of the input are identified. Each identified context feature is compared with the identified human-object interaction. An importance of the identified context feature is determined for the identified human-object interaction. The context feature is fused with the identified human-object interaction when the importance is greater than a threshold.
1. A processor-implemented method for classifying an input, the processor implemented method being performed by one or more processors and comprising:
identifying a human-object interaction in the input;
identifying context features of the input;
comparing each identified context feature with the identified human-object interaction;
determining an importance score of the identified context feature for the identified human-object interaction based on the comparing;
fusing the context feature with the identified human-object interaction to obtain a fused representation responsive to the importance score being greater than a threshold; and
classifying the human-object interaction based on the fused representation.
2. The processor-implemented method of claim 1 , in which the context features comprise one or more of clothing, human body part features, human-object spatial relations, scene locality, scene geometry, or co-occurring objects.
3. The processor-implemented method of claim 1 , further comprising determining a probability of a verb and noun pair based on the fused representation of the human-object interaction and the context features.
4. The processor-implemented method of claim 1 , further comprising adding noise during a training forward pass to robustify the importance score determination.
5. The processor-implemented method of claim 4 , further comprising removing the noise during a training backward pass.
6. The processor-implemented method of claim 1 , in which a classifier for classifying the input is robust to horizontal and vertical flipping.
7. The processor-implemented method of claim 1 , further comprising jittering a location of one or more of a human bounding box or object bounding box during training.
8. An apparatus, comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
identify a human-object interaction in an input;
identify context features of the input;
perform a comparison of each identified context feature with the identified human-object interaction;
determine an importance score of the identified context feature for the identified human-object interaction based on the comparison;
fuse the context feature with the identified human-object interaction to obtain a fused representation responsive to the importance score being greater than a threshold; and
classify the human-object interaction based on the fused representation.
9. The apparatus of claim 8 , in which the context features comprise one or more of clothing, human-object spatial relations, scene locality, scene geometry, or co-occurring objects.
10. The apparatus of claim 8 , in which the at least one processor is further configured to determine a probability of a verb and noun pair based on the fused representation of the human-object interaction and the context features.
11. The apparatus of claim 8 , in which the at least one processor is further configured to add noise during a training forward pass to robustify the importance score determination.
12. The apparatus of claim 11 , in which the at least one processor is further configured to remove the noise during a training backward pass.
13. The apparatus of claim 8 , in which a classifier for classifying the input is robust to horizontal and vertical flipping.
14. The apparatus of claim 8 , in which the at least one processor is further configured to jitter a location of one or more of a human bounding box or object bounding box during training.
15. An apparatus, comprising:
means for identifying a human-object interaction in an input;
means for identifying context features of the input;
means for performing a comparison of each identified context feature with the identified human-object interaction;
means for determining an importance score of the identified context feature for the identified human-object interaction based on the comparison;
means for fusing the context feature with the identified human-object interaction to obtain a fused representation responsive to the importance score being greater than a threshold; and
means for classifying the human-object interaction based on the fused representation.
16. The apparatus of claim 15 , in which the context features comprise one or more of clothing, human body part features, human-object spatial relations, scene locality, scene geometry, or co-occurring objects.
17. The apparatus of claim 15 , further comprising means for determining a probability of a verb and noun pair based on the fused representation of the human-object interaction and the context features.
18. The apparatus of claim 15 , further comprising means for adding noise during a training forward pass to robustify the importance score determination.
19. The apparatus of claim 18 , further comprising means for removing the noise during a training backward pass.
20. The apparatus of claim 15 , in which a classifier for classifying the input is robust to horizontal and vertical flipping.
21. The apparatus of claim 15 , further comprising means for jittering a location of one or more of a human bounding box or object bounding box during training.
22. A non-transitory computer-readable medium having program code recorded thereon, the program code executed by a processor and comprising:
program code to identify a human-object interaction in an input;
program code to identify context features of the input;
program code to perform a comparison of each identified context feature with the identified human-object interaction;
program code to determine an importance score of the identified context feature for the identified human-object interaction based on the comparison;
program code to fuse the context feature with the identified human-object interaction to obtain a fused representation responsive to the importance score being greater than a threshold; and
program code to classify the human-object interaction based on the fused representation.
23. The non-transitory computer-readable medium of claim 22 , in which the context features comprises one or more of clothing, human-object spatial relations, scene locality, scene geometry, or co-occurring objects.
24. The non-transitory computer-readable medium of claim 22 , further comprising program code to determine a probability of a verb and noun pair based on the fused representation of the human-object interaction and the context features.
25. The non-transitory computer-readable medium of claim 22 , further comprising program code to add noise during a training forward pass to robustify the importance score determination.
26. The non-transitory computer-readable medium of claim 25 , further comprising program code to remove the noise during a training backward pass.
27. The non-transitory computer-readable medium of claim 22 , in which a classifier for classifying the input is robust to horizontal and vertical flipping.
28. The non-transitory computer-readable medium of claim 22 , further comprising program code to jitter a location of one or more of a human bounding box or object bounding box during training.