Dummy prototypical networks for few-shot open-set keyword spotting
Systems and techniques are provided for processing audio data. For example, a dummy prototypical network may be used to perform few-shot open-set keyword spotting (FSOS-KWS). A process can include determining one or more prototype representations based on a plurality of support samples associated with one or more classes. Each prototype representation may be associated with one of the class(es). A dummy prototype representation can be determined in a same learned metric space as the prototype representations. One or more distance metrics can be determined for each query sample of one or more query samples. The distance metrics may be based on the prototype representations and the dummy prototype representation. Each query sample can be classified based on the distance metrics. Each query sample may be classified into one of the class(es) associated with the prototype representations or into an open-set class associated with the dummy prototype representation.
1. A method for processing one or more data samples, comprising:
obtaining a support set corresponding to a few shot learning (FSL) inference episode associated with a first machine learning network, wherein the support set includes a plurality of labeled support samples associated with one or more known classes of the support set;
determining a set of prototype representations based on the plurality of labeled support samples, wherein the set of prototype representations is determined in a learned embedding space of the first machine learning network, and wherein each prototype representation of the set of prototype representations corresponds to a respective known class of the one or more known classes;
generating, based on processing the set of prototype representations using a second machine learning network, a dummy prototype representation corresponding to a predicted dummy class for the FSL inference episode, wherein the second machine learning network generates the dummy prototype representation in the same learned embedding space used to determine the set of prototype representations;
determining one or more distance metrics for each query sample of one or more query samples corresponding to the FSL inference episode, wherein the one or more distance metrics are determined based on an augmented prototype set comprising the set of prototype representations and the dummy prototype representation; and
classifying each query sample of the one or more query samples based on the one or more distance metrics between each query sample and the augmented prototype set, wherein each query sample is classified into one of the one or more known classes associated with the set of prototype representations or is classified into an open-set class associated with the dummy prototype representation, and wherein the open-set class is not included in the one or more known classes.
2. The method of claim 1 , wherein classifying each query sample based on the one or more distance metrics includes:
determining a probability distribution over the one or more known classes and the open-set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric;
classifying, based on the probability distribution, each query sample into one of the one or more known classes or into the open-set class; and
performing open-set rejection (OSR) based on a set of classified query samples classified into the open-set class associated with the dummy prototype representation.
3. The method of claim 1 , wherein the one or more known classes are keyword classes, and wherein each prototype representation is determined using a plurality of support samples associated with a keyword class.
4. The method of claim 3 , further comprising:
generating, using a neural network encoder of the first machine learning network, a plurality of embeddings for the plurality of support samples associated with the keyword class, the plurality of embeddings including an embedding for each support sample of the plurality of support samples; and
determining each prototype representation based on an average of the plurality of embeddings generated for the plurality of support samples.
5. The method of claim 4 , wherein the learned embedding space is a learned embedding space of the neural network encoder.
6. The method of claim 3 , wherein the keyword classes comprise a closed-set of known classes.
7. The method of claim 1 , wherein the second machine learning network is a dummy generator network, wherein:
the dummy generator network includes one or more linear layers and a maxpool layer;
the one or more linear layers receive as input the set of prototype representations; and
the maxpool layer receives as input an output of the one or more linear layers.
8. The method of claim 7 , wherein generating the dummy prototype representation further comprises:
generating, using the dummy generator network, a plurality of dummy prototypes; and
determining the dummy prototype representation by selecting one of the plurality of dummy prototypes based on a Gumbel softmax.
9. The method of claim 1 , wherein:
the one or more query samples include speech audio data; and
the plurality of labeled support samples includes speech audio data labeled with a corresponding one of the one or more known classes for each labeled support sample of the plurality of labeled support samples.
10. The method of claim 1 , wherein the dummy prototype representation is an episode-known dummy prototype representation corresponding to an open set of unknown classes specific for the FSL inference episode.
11. An apparatus for processing one or more data samples, comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
obtain a support set corresponding to a few shot learning (FSL) inference episode associated with a first machine learning network, wherein the support set includes a plurality of labeled support samples associated with one or more known classes of the support set;
determine a set of prototype representations based on the plurality of labeled support samples, wherein the set of prototype representations is determined in a learned embedding space of the first machine learning network, and wherein each prototype representation of the set of prototype representations corresponds to a respective known class of the one or more known classes;
generate, based on processing the set of prototype representations using a second machine learning network, a dummy prototype representation corresponding to a predicted dummy class for the FSL inference episode, wherein the second machine learning network generates the dummy prototype representation in the same learned embedding space used to determine the set of prototype representations;
determine one or more distance metrics for each query sample of one or more query samples corresponding to the FSL inference episode, wherein the one or more distance metrics are determined based on an augmented prototype set comprising the set of prototype representations and the dummy prototype representation; and
classify each query sample of the one or more query samples based on the one or more distance metrics between each query sample and the augmented prototype set, wherein each query sample is classified into one of the one or more known classes associated with the set of prototype representations or is classified into an open-set class associated with the dummy prototype representation, and wherein the open-set class is not included in the one or more known classes.
12. The apparatus of claim 11 , wherein, to classify each query sample based on the one or more distance metrics, the at least one processor is configured to:
determine a probability distribution over the one or more known classes and the open-set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric;
classify, based on the probability distribution, each query sample into one of the one or more known classes or into the open-set class; and
perform open-set rejection (OSR) based on a set of classified query samples classified into the open-set class associated with the dummy prototype representation.
13. The apparatus of claim 11 , wherein the one or more known classes are keyword classes and the at least one processor is configured to determine each prototype representation using a plurality of support samples associated with a keyword class.
14. The apparatus of claim 13 , wherein the at least one processor is further configured to:
generate, using a neural network encoder of the first machine learning network, a plurality of embeddings for the plurality of support samples associated with the keyword class, the plurality of embeddings including an embedding for each support sample of the plurality of support samples; and
determine each prototype representation based on an average of the plurality of embeddings generated for the plurality of support samples.
15. The apparatus of claim 14 , wherein the learned embedding space is a learned embedding space of the neural network encoder.
16. The apparatus of claim 13 , wherein the keyword classes comprise a closed-set of known classes.
17. The apparatus of claim 11 , wherein the second machine learning network is a dummy generator network, wherein:
the dummy generator network includes one or more linear layers and a maxpool layer;
the one or more linear layers receive as input the set of prototype representations; and
the maxpool layer receives as input an output of the one or more linear layers.
18. The apparatus of claim 17 , wherein, to generate the dummy prototype representation, the at least one processor is configured to:
generate, using the dummy generator network, a plurality of dummy prototypes; and
determine the dummy prototype representation by selecting one of the plurality of dummy prototypes based on a Gumbel softmax.
19. The apparatus of claim 11 , wherein:
the one or more query samples include speech audio data; and
the plurality of labeled support samples includes speech audio data labeled with a corresponding one of the one or more known classes for each labeled support sample of the plurality of labeled support samples.
20. The apparatus of claim 11 , wherein the dummy prototype representation is an episode-known dummy prototype representation corresponding to an open set of unknown classes specific for the FSL inference episode.
21. A non-transitory computer-readable storage medium comprising instructions stored thereon which, when executed by at least one processor, causes the at least one processor to:
obtain a support set corresponding to a few shot learning (FSL) inference episode associated with a first machine learning network, wherein the support set includes a plurality of labeled support samples associated with one or more known classes of the support set;
determine a set of prototype representations based on the plurality of labeled support samples, wherein the set of prototype representations is determined in a learned embedding space of the first machine learning network, and wherein each prototype representation of the set of prototype representations corresponds to a respective known class of the one or more known classes;
generate, based on processing the set of prototype representations using a second machine learning network, a dummy prototype representation corresponding to a predicted dummy class for the FSL inference episode, wherein the second machine learning network generates the dummy prototype representation in the same learned embedding space used to determine the set of prototype representations;
determine one or more distance metrics for each query sample of one or more query samples corresponding to the FSL inference episode, wherein the one or more distance metrics are determined based on an augmented prototype set comprising the set of prototype representations and the dummy prototype representation; and
classify each query sample of the one or more query samples based on the one or more distance metrics between each query sample and the augmented prototype set, wherein each query sample is classified into one of the one or more known classes associated with the set of prototype representations or is classified into an open-set class associated with the dummy prototype representation, and wherein the open-set class is not included in the one or more known classes.
22. The non-transitory computer-readable storage medium of claim 21 , wherein, to classify each query sample based on the one or more distance metrics, the instructions cause the at least one processor to:
determine a probability distribution over the one or more known classes and the open-set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric;
classify, based on the probability distribution, each query sample into one of the one or more known classes or into the open-set class; and
perform open-set rejection (OSR) based on a set of classified query samples classified into the open-set class associated with the dummy prototype representation.
23. The non-transitory computer-readable storage medium of claim 21 , wherein the one or more known classes are keyword classes and the at least one processor is configured to determine each prototype representation using a plurality of support samples associated with a keyword class.
24. The non-transitory computer-readable storage medium of claim 23 , wherein the instructions further cause the at least one processor to:
generate, using a neural network encoder of the first machine learning network, a plurality of embeddings for the plurality of support samples associated with the keyword class, the plurality of embeddings including an embedding for each support sample of the plurality of support samples; and
determine each prototype representation based on an average of the plurality of embeddings generated for the plurality of support samples.
25. The non-transitory computer-readable storage medium of claim 24 , wherein the learned embedding space is a learned embedding space of the neural network encoder.
26. The non-transitory computer-readable storage medium of claim 23 , wherein the keyword classes comprise a closed-set of known classes.
27. The non-transitory computer-readable storage medium of claim 21 , wherein the second machine learning network is a dummy generator network, wherein:
the dummy generator network includes one or more linear layers and a maxpool layer;
the one or more linear layers receive as input the set of prototype representations; and
the maxpool layer receives as input an output of the one or more linear layers.
28. The non-transitory computer-readable storage medium of claim 27 , wherein, to generate the dummy prototype representation, the instructions cause the at least one processor to:
generate, using the dummy generator network, a plurality of dummy prototypes; and
determine the dummy prototype representation by selecting one of the plurality of dummy prototypes based on a Gumbel softmax.
29. The non-transitory computer-readable storage medium of claim 21 , wherein:
the one or more query samples include speech audio data; and
the plurality of labeled support samples includes speech audio data labeled with a corresponding one of the one or more known classes for each labeled support sample of the plurality of labeled support samples.
30. The non-transitory computer-readable storage medium of claim 21 , wherein the dummy prototype representation is an episode-known dummy prototype representation corresponding to an open set of unknown classes specific for the FSL inference episode.