Granted Patent B2

US 12,346,667 · App. 17/952,817 · Granted Jul 1, 2025

Inventor: Olga Vechtomova (Baden, CA)

G06F40/56G10H1/0008G10L25/18G10L25/30G10H2210/031G10H2250/311

Abstract

Claims (70)

1. A method of generating lyrics, the method comprising:

obtaining, from an encoding portion of a first autoencoder, a representation of a time-limited audio recording;

sampling, from a second distribution, a text vector;

generating an output lyric line by decoding a text decoder input vector that is based, at least in part, on the representation and the text vector;

wherein the decoding uses a decoding portion of a second autoencoder, wherein the second autoencoder is a latent variable model autoencoder;

wherein the decoding portion of the second autoencoder has been trained to generate reconstructed output lyric lines based, at least in part, on input that includes:

a first latent vector from a latent space of the first autoencoder, the first autoencoder trained with spectrogram input of known musical works; and

a second latent vector sampled from a distribution in a latent space of the second autoencoder, the second autoencoder trained with lyric input of the known musical works corresponding to the spectrogram input of known musical works;

wherein the distribution in the latent space of the second autoencoder is encoded lyric input corresponding to a spectrogram input encoded to lead to the first latent vector.

2. The method of claim 1 , wherein the first autoencoder comprises a variational autoencoder.

3. The method of claim 2 , further comprising:

encoding the spectrogram to produce a first distribution; and

sampling, from the first distribution, the representation.

4. The method of claim 1 , wherein the second autoencoder comprises a variational autoencoder.

5. The method of claim 1 , wherein the second autoencoder comprises a Wasserstein autoencoder.

6. The method of claim 1 , wherein the second autoencoder comprises an Adversarially Regularized autoencoder.

7. The method of claim 1 , wherein the encoding portion of the first autoencoder comprises a convolutional neural network.

8. The method of claim 1 , wherein the decoding portion of the second autoencoder comprises a long short term memory network.

9. The method of claim 1 , wherein the decoding portion of the second autoencoder comprises gated recurrent units.

10. The method of claim 1 , wherein the decoding portion of the second autoencoder comprises a transformer-based network.

11. The method of claim 1 , wherein the second distribution comprises a prior text distribution.

12. The method of claim 1 , wherein the second distribution comprises a posterior text distribution.

13. The method of claim 1 , further comprising obtaining a spectrogram, wherein the spectrogram is representative of the time-limited audio recording.

14. A method of generating lyrics, the method comprising:

obtaining a spectrogram, where the spectrogram is representative of a time-limited audio recording;

encoding the spectrogram to, thereby, produce a first distribution, the encoding using an encoding portion of a first trained variational autoencoder;

sampling, from the first distribution, an inference spectrogram latent code;

generating, by providing the inference spectrogram latent code as input to a trained Generative Adversarial Network, an inference text latent code;

generating an output lyric line by decoding a text decoder input vector that is based, at least in part, on the inference spectrogram latent code and the inference text latent code;

wherein the decoding uses a decoding portion of a variational autoencoder;

wherein the variational autoencoder has been trained to generate reconstructed output lyric lines;

wherein training the variational autoencoder includes:

encoding a training spectrogram to, thereby, produce a training distribution, the encoding using the encoding portion of the first variational autoencoder;

sampling, from the training distribution, a training spectrogram latent code;

encoding a training input to, thereby, obtain a second distribution, the encoding using an encoding portion of the variational autoencoder, the training input including:

a lyric line corresponding to the training spectrogram; and

the training spectrogram latent code; and

sampling, from the second distribution, a training text latent code;

providing, as input to the decoder portion of the variational autoencoder, a training text decoder input vector that is based, at least in part, on the training spectrogram latent code and the training text latent code.

15. The method of claim 14 , wherein the variational autoencoder comprises a conditional variational autoencoder.

16. A method of generating lyrics, the method comprising:

obtaining a spectrogram, where the spectrogram is representative of a time-limited audio recording;

encoding the spectrogram to, thereby, produce a first distribution, the encoding using an encoding portion of a first trained variational autoencoder;

sampling, from the first distribution, an inference spectrogram latent vector;

sampling, from a second distribution, an inference text latent vector, wherein a location, in a text latent space of a second trained conditional variational autoencoder, of the second distribution corresponds to a location, in a latent space of the first trained variational autoencoder, of the first distribution;

generating an output lyric line by decoding a text decoder input vector that is based, at least in part, on the inference spectrogram latent vector and the inference text latent vector;

wherein the decoding uses a decoding portion of the conditional variational autoencoder;

wherein the conditional variational autoencoder has been trained to generate reconstructed output lyric lines;

wherein training the conditional variational autoencoder includes:

encoding a training spectrogram to, thereby, produce a training distribution, the encoding using the encoding portion of the first variational autoencoder;

sampling, from the training distribution, a training spectrogram latent vector;

encoding a training input to, thereby, obtain a second distribution, the encoding using an encoding portion of the conditional variational autoencoder, the training input including:

a lyric line corresponding to the training spectrogram; and

the training spectrogram latent vector; and

sampling, from the second distribution, a training text latent vector;

providing, as input to the decoder portion of the conditional variational autoencoder, a training text decoder input vector that is based, at least in part, on the training spectrogram latent vector and the training text latent vector.

17. The method of claim 1 , further comprising:

receiving a plurality of known songs that include lyrics;

dividing each song of the plurality of known songs into a plurality of intervals;

training the first autoencoder to generate the representation from an input spectrogram derived from an interval among the plurality of intervals, wherein the training the first autoencoder causes generation of the latent space of the first autoencoder including a plurality of first distributions, where each first distribution in the plurality of first distributions corresponds to an interval among the plurality of intervals; and

training the second autoencoder to generate a reconstructed lyric line from an input lyric line derived from an interval among the plurality of intervals, wherein the training the second autoencoder causes generation of the latent space of the second autoencoder including a plurality of second distributions, where each second distribution in the plurality of second distributions corresponds to an interval among the plurality of intervals;

wherein, during the training the second variational autoencoder, a decoder portion of the second variational autoencoder is configured to generate the reconstructed lyric line based on input that includes:

the first latent vector selected from a first distribution in the latent space of the first autoencoder, the first distribution corresponding to a given interval, and

the text vector sampled from the second distribution in the latent space of the second autoencoder, the second distribution corresponding to the given interval.

18. The method of claim 1 , further comprising:

receiving a plurality of known songs that include lyrics;

dividing each song of the plurality of known songs into a plurality of intervals;

training the first autoencoder to generate a reconstructed spectrogram from an input spectrogram derived from an interval among the plurality of intervals, wherein the training the first autoencoder causes generation of a first latent space including a plurality of first distributions, where each first distribution in the plurality of first distributions corresponds to an interval among the plurality of intervals; and

training the second autoencoder to generate a reconstructed lyric line from an input lyric line derived from a particular interval among the plurality of intervals in combination with a first latent vector sampled from a particular first distribution in the plurality of first distributions, wherein the particular first distribution corresponds to the particular interval and wherein the training the second autoencoder causes generation of a second latent space including a plurality of second distributions, where each second distribution in the plurality of second distributions corresponds to an interval among the plurality of intervals;

wherein, during the training the second variational autoencoder, a decoder portion of the second variational autoencoder is configured to generate the reconstructed lyric line based on input that includes the first latent vector and a second latent vector selected from a second distribution in the second latent space, the second distribution corresponding to the particular interval.

Continuity (2)

References Cited (42)