System and method for improved few-shot object detection using a dynamic semantic network
Disclosed herein is an improved few-shot detector which utilizes a dynamic semantic network which takes as input a language feature and generates trainable parameters for a visual network. The visual network takes a visual feature as input and generates a classification and localization of an object.
1. A few-shot object detector comprising:
a visual network with trainable parameters producing an output based on an input of a visual feature; and
a dynamic semantic network which accepts as input a language feature representing a semantic representation of a class and outputs a class-specific parameter for the visual network;
wherein the dynamic semantic network includes a dynamic relation graph for building direct connections between base classes and novel classes using non-visual knowledge of the base classes.
2. The few-shot detector of claim 1 wherein the language feature input to the dynamic semantic network represents a language representation of a class for which the visual network is trained to detect.
3. The few-shot detector of claim 2 further comprising a loss function generating a gradient for backpropagating to the visual network.
4. The few-shot detector of claim 3 wherein the visual network backpropagates the gradients to the dynamic semantic network.
5. The few-shot detector of claim 4 wherein the visual network computes partial derivatives of the gradients using chain rules before backpropagating the gradients to the dynamic semantic network.
6. The few-shot detector of claim 5 wherein the dynamic semantic network uses the gradient received from the visual network to update the trainable parameters of the dynamic semantic network.
7. The few-shot detector of claim 6 wherein the visual network comprises a classification sub-network and a localization sub-network.
8. The few-shot detector of claim 7 wherein the dynamic semantic network updates trainable parameters of both the classification sub-network and the localization sub-network.
9. The few-shot detector of claim 1 wherein the visual network is trained on a dataset comprising many instances of base class objects and few instances of novel class objects.
10. A system comprising:
a processor; and
memory, storing software that, when executed by the processor, implements the few-shot detector of claim 1 .
11. A method comprising:
training a visual network with trainable parameters to produce an output based on an input of a visual feature; and
training a dynamic semantic network to accept as input a language feature representing a semantic representation of a class and to output a class-specific parameter for the visual network;
wherein the dynamic semantic network includes a dynamic relation graph for building direct connections between base classes and novel classes using non-visual knowledge of the base classes.
12. The method of claim 11 wherein the language feature input to the dynamic semantic network represents a language representation of a class for which the visual network is trained to detect.
13. The method of claim 12 wherein the visual network:
receives backpropagated gradients from a loss function.
14. The method of claim 13 wherein the visual network:
backpropagates the gradients to the dynamic semantic network.
15. The method of claim 14 wherein the visual network:
computes partial derivatives of the gradients using chain rules before backpropagating the gradients to the dynamic semantic network.
16. The method of claim 15 wherein the dynamic semantic network:
uses the gradients received from the visual network to update the trainable parameters of the dynamic semantic network.
17. The method of claim 16 wherein the visual network comprises a classification sub-network and a localization sub-network.
18. The method of claim 7 wherein the dynamic semantic network:
generates the trainable parameters of both the classification sub-network and the localization sub-network.
19. The method of claim 11 wherein the visual network is trained on a dataset comprising many instances of base class objects and few instances of novel class objects.
20. A system comprising:
a processor; and
memory, storing software that, when executed by the processor, performs the method of claim 11 .
21. The method of claim 1 wherein the dynamic relation graph adapts word embeddings for a vision domain.
22. The method of claim 11 wherein the dynamic relation graph adapts word embeddings for a vision domain.