Transmodal translation of feature vectors to audio for assistive devices
A computer implemented method is provided that includes embedding a received signal in a first modality, re-embedding the embedded received signal of the first modality into a signal of a second modality, and generating an output in the second modality, and based on the output, rendering a signal in the second modality that is configured to be sensed, wherein the embedding, re-embedding and generating applies a model that is trained by performing an adversarial learning operation associated with discriminating actual examples of the target distribution from the generated output, and performing a metric learning operation associated with generating the output having perceptual distances.
1. A computer implemented method, comprising:
embedding a received signal in a first modality;
re-embedding the embedded received signal of the first modality into a signal of a second modality, and generating an output in the second modality; and
based on the output, rendering a signal in the second modality that is configured to be sensed, wherein the embedding, re-embedding and generating applies a model that is trained by performing an adversarial learning operation associated with discriminating actual examples of the target distribution from the generated output, and performing a metric learning operation associated with generating the output having perceptual distances.
2. The computer implemented method of claim 1 , wherein the embedding is performed by an encoder that applies a feature embedding model.
3. The computer implemented method of claim 1 , wherein the re-embedding is performed by a re-embedding network.
4. The computer implemented method of claim 1 , wherein the performing the adversarial learning comprises providing the generated output to a discriminator network that discriminates between the generated output and the actual version of the output, to generate a discriminator loss.
5. The computer implemented method of claim 1 , wherein the performing the learning metric comprises applying a mel-frequency cepstral (MFC) transform, to generate a metric loss function associated with determining the perceptual distances.
6. The computer implemented method of claim 1 , wherein the first modality is visual and the second modality is audio.
7. A non-transitory computer readable medium configured to execute machine-readable instructions on a processor, the instructions comprising:
embedding a received signal in a first modality;
re-embedding the embedded received signal of the first modality into a signal of a second modality, and generating an output in the second modality; and
based on the output, rendering a signal in the second modality that is configured to be sensed, wherein the embedding, re-embedding and generating applies a model that is trained by performing an adversarial learning operation associated with discriminating actual examples of the target distribution from the generated output, and performing a metric learning operation associated with generating the output having perceptual distances.
8. The non-transitory computer readable medium of claim 7 , wherein the embedding is performed by an encoder that applies a feature embedding model.
9. The non-transitory computer readable medium of claim 7 , wherein the re-embedding is performed by a re-embedding network.
10. The non-transitory computer readable medium of claim 7 , wherein the performing the adversarial learning comprises providing the generated output to a discriminator network that discriminates between the generated output and the actual version of the output, to generate a discriminator loss.
11. The non-transitory computer readable medium of claim 7 , wherein the performing the learning metric comprises applying a mel-frequency cepstral (WC) transform, to generate a metric loss function associated with determining the perceptual distances.
12. The non-transitory computer readable medium of claim 7 , wherein the first modality is visual and the second modality is audio.
13. An apparatus, comprising:
an input device configured to receive information having a first modality;
an output device configured to output the information having a second modality; and
a processor that receives the information having the first modality and generates the information having the second modality, the processor configured to:
embed a received signal in a first modality;
re-embed the embedded received signal of the first modality into a signal of a second modality, and generate an output in the second modality; and
based on the output, render a signal in the second modality that is configured to be sensed, wherein the processor configured to embed, re-embed and generate applies a model that is trained by performing an adversarial learning operation associated with discriminating actual examples of the target distribution from the generated output, and performing a metric learning operation associated with generating the output having perceptual distances.
14. The apparatus of claim 13 , wherein the input device comprises a camera and the output device comprises a speaker or a headphone.
15. The apparatus of claim 13 , wherein the first modality is visual and the second modality is audio.
16. The apparatus of claim 13 , wherein the input device and output device are mounted on a wearable device.
17. The apparatus of claim 16 , wherein the wearable device comprises glasses.
18. The apparatus of claim 13 , wherein the processor is configured to embed by an encoder that applies a feature embedding model, and to re-embed by a re-embedding network.
19. The apparatus of claim 13 , wherein the performing the adversarial learning comprises providing the generated output to a discriminator network that discriminates between the generated output and the actual version of the output, to generate a discriminator loss, and the performing the learning metric comprises applying a mel-frequency cepstral (MFC) transform, to generate a metric loss function associated with determining the perceptual distances.
20. The apparatus of claim 13 , wherein annotated data is not required to learn the mapping between the first modality and the second modality.