Systems and methods for cross-modal signal inference using audio signals
Systems and methods for converting a primary one-dimensional signal into a secondary one-dimensional signal of another modality. The primary signal is spliced into a plurality of consecutive frames. A first linear transformation transforms the frames into corresponding vectors. Positional encodings are provided on the vectors to encode relative positional information associated with each sample within each frame. A multi-head self-attention machine-learning model compares relative importance of the samples within each vector to each other in that vector to yield high-level representation vectors. A second linear transformation transforms the high-level representation vectors into corresponding secondary signal frames. The secondary signal frames are concatenated into a reconstructed one-dimensional secondary signal having a different modality than the primary signal.
1. A method of transforming an audio signal into a secondary signal of another modality, the method comprising:
receiving an audio signal generated from a microphone;
splicing the audio signal into a plurality of frames, each frame having a number of samples of audio data;
executing a first linear transformation to transform the frames into corresponding vectors;
executing positional encoding on the vectors to encode relative positional information associated with each sample within the vectors;
executing a transformer encoder on the vectors with the encoded positional information, wherein the transformer encoder has a multi-head self-attention mechanism configured to compare relative importance of the vectors to each other to yield high-level representation vectors;
executing a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames; and
concatenating the corresponding secondary signal frames into a reconstructed one-dimensional secondary signal having a different modality than the audio signal.
2. The method of claim 1 , wherein each frame has an identical size τ.
3. The method of claim 1 , wherein the transformer encoder has an add and normalization feature configured to add an output of a layer of the transformer encoder to an input of the layer and normalize values in the output of the layer.
4. The method of claim 1 , wherein the multi-head self-attention mechanism includes an attention function that maps a query and a set of pairs of keys and values to an output, wherein the query, the keys, the values, and the output are all vectors.
5. The method of claim 4 , wherein the multi-head self-attention mechanism is configured to compute the output vector as a weighted sum of the values.
6. The method of claim 5 , wherein weights assigned to each value are computed by a compatibility function of the query with a corresponding one of the keys.
7. The method of claim 1 , wherein the multi-head self-attention mechanism is configured to, for each sample, compute a score for that sample representing the relative importance of that sample relative to the other samples within that frame.
8. The method of claim 7 , wherein the scores are associated with how much each sample should contribute to the output of the transformer encoder.
9. The method of claim 1 , wherein the secondary signal is a torque signal or a vibration signal.
10. A system for converting a primary one-dimensional signal into a secondary one-dimensional signal of another modality, the system comprising:
a processor programmed to execute instructions stored in memory to:
splice a primary signal into a plurality of consecutive frames;
perform a first linear transformation to transform the frames into corresponding vectors;
execute positional encoding on the vectors to encode relative positional information associated with each sample, wherein the relative positional information is associated with a sequential position of each sample within its respective frame;
execute a multi-head self-attention mechanism configured to compare relative importance of the samples to each other within its respective frame to yield high-level representation vectors;
perform a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames; and
concatenating the secondary signal frames into a reconstructed one-dimensional secondary signal having a different modality than the primary signal.
11. The system of claim 10 , wherein each frame has an identical size.
12. The system of claim 10 , wherein the multi-head self-attention mechanism has an add and normalization feature configured to add an output of a layer of the transformer encoder to an input of the layer and normalize values in the output of the layer.
13. The system of claim 10 , wherein the multi-head self-attention mechanism includes an attention function that maps a query and a set of pairs of keys and values to an output, wherein the query, the keys, the values, and the output are all vectors.
14. The system of claim 13 , wherein the multi-head self-attention mechanism is configured to compute the output vector as a weighted sum of the values.
15. The system of claim 14 , wherein weights assigned to each value are computed by a compatibility function of the query with a corresponding one of the keys.
16. The system of claim 10 , wherein the multi-head self-attention mechanism is configured to, for each sample within a respective one of the frames, compute a score for that sample representing the relative importance of that sample relative to the other samples in that frame.
17. The system of claim 16 , wherein the scores are associated with how much each sample should contribute to the output of the transformer encoder.
18. The system of claim 10 , wherein the primary signal is a sound signal.
19. A computer-controlled machine comprising the system of claim 10 , wherein the computer-controlled machine further comprises an actuator configured to control an operation of the computer-controlled machine based on an output of the system.
20. A computer-controlled machine comprising:
at least one microphone configured to generate an audio signal;
a control system configured to predict an operational characteristic of the computer-controlled machine by translating the audio signal into a secondary one-dimensional signal representative of the operational characteristic, the control system configured to:
splice the audio signal into a plurality of frames,
execute a first linear transformation to transform the frames into corresponding vectors;
execute positional encoding on the vectors to encode relative positional information associated with each sample;
execute a transformer encoder on the vectors with the encoded positional information, wherein the transformer encoder has a multi-head self-attention mechanism configured to compare relative importance of the samples to each other within the respective vectors to yield high-level representation vectors;
execute a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames; and
concatenate the corresponding secondary signal frames into a reconstructed one-dimensional secondary signal having a different modality than the audio signal.