Automatic audio captioning

1. A computer-implemented method for audio captioning, comprising:

receiving a raw audio waveform including a non-speech sound;

extracting, in one or more processors configured to implement a recurrent neural network (RNN) acoustic model, relevant features from the raw audio waveform using the RNN acoustic model;

processing the relevant features by an attention decoder to generate an attention weight value for one or more timesteps during the non-speech sound; and

successively generating a character for each attention weight value to produce a discrete sequence of characters represented in a natural language, wherein the discrete sequence of characters comprises a caption that describes the non-speech sound.

2. The computer-implemented method of claim 1 , wherein the raw audio waveform further includes a speech sound and the caption also comprises text corresponding to the spoken language within the speech sound.

3. The computer-implemented method of claim 1 , further comprising a RNN language model that receives the sequence of characters and at least one additional sequence of characters and selects the sequence of characters as the caption based on a probability value corresponding to validity of the sequence of characters in a target language.

4. The computer-implemented method of claim 1 , further comprising training the RNN acoustic model using training audio waveforms, each training audio waveform comprising a combination of a first non-speech sound and a second sound that overlaps at least partially in time with the first non-speech sound.

5. The computer-implemented method of claim 4 , wherein the combination corresponds to a first caption.

6. The computer-implemented method of claim 5 , wherein a length of the overlap of the first non-speech sound and the second sound in each training audio waveform corresponding to the first caption varies among different training audio waveforms.

7. The computer-implemented method of claim 4 , wherein the second sound is either a non-speech sound or a speech sound.

8. The computer-implemented method of claim 4 , wherein the second sound is noise samples drawn from white noise, music, or television clips.

9. The computer-implemented method of claim 1 , wherein the RNN acoustic model processes the relevant features and generates a context vector of activations for each timestep of the raw audio waveform including the non-speech sound.

10. The computer-implemented method of claim 9 , wherein a number of activations in a first context vector is different than a number of activations in a second context vector.

11. The computer-implemented method of claim 9 , wherein the attention decoder performs a continuous associative lookup operation over the context vector and at least one additional context vector to generate the attention weight values for each timestep where the non-speech sound is present.

12. The computer-implemented method of claim 1 , wherein a first character generated by a RNN language model for a previous timestep of the one or more timesteps is used to determine an end of the discrete sequence of characters.

13. An audio captioning system, comprising:

one or more processors configured to implement a recurrent neural network (RNN) acoustic model that receives a raw audio waveform including a non-speech sound and extracts relevant features from the raw audio waveform; and

an attention decoder that is configured to:

process the relevant features to generate an attention weight value for one or more timesteps during the non-speech sound; and

successively generate a character for each attention weight value to produce a discrete sequence of characters represented in a natural language, wherein the discrete sequence of characters comprises a caption that describes the non-speech sound.

14. The audio captioning system of claim 13 , wherein the RNN acoustic model includes a context unit that processes the relevant features and generates a context vector of activations for each timestep of the raw audio waveform including the non-speech sound.

15. The audio captioning system of claim 14 , wherein the attention decoder performs a continuous associative lookup operation over the context vector and at least one additional context vector to generate the attention weight values for each timestep where the non-speech sound is present.

16. A non-transitory computer-readable media storing computer instructions for generating captions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

receiving a raw audio waveform including a non-speech sound;

extracting, by the one or more processors configured to implement a recurrent neural network (RNN) acoustic model, relevant features from the raw audio waveform using the RNN acoustic model;

processing the relevant features by an attention decoder to generate an attention weight value for one or more timesteps during the non-speech sound; and

successively generating a character for each attention weight value to produce a discrete sequence of characters represented in a natural language, wherein the discrete sequence of characters comprises a caption that describes the non-speech sound.

17. The computer-implemented method of claim 12 , wherein the RNN language model corrects a mistake in the discrete sequence of characters to produce the caption.

18. The computer-implemented method of claim 9 , wherein the context vector is generated before a last timestep of the one or more timesteps during the non-speech sound.

19. The computer-implemented method of claim 9 , wherein the context vector tags each timestep of the audio clip to indicate whether the non-speech sound is present during the timestep.

20. The audio captioning system of claim 13 , further comprising a RNN language model coupled between the RNN acoustic model and the decoder, wherein a first character generated by the RNN language model for a previous timestep of the one or more timesteps is used to determine an end of the discrete sequence of characters.