Synchronized sound generation from videos

1. A computing device comprising:

a processor;

a network interface coupled to the processor to enable communication over a network;

a storage device for content and programming coupled to the processor;

a sound engine stored in the storage device, wherein an execution of the sound engine by the processor configures the computing device to perform acts comprising:

receiving a video feed;

dividing the video feed into a sequence of video segments;

for each video segment:

extracting visual features of the video segment;

generating a predicted spectrogram based on the extracted visual features; and

generating a synthetic audio waveform from the predicted spectrogram; and

concatenating all synthetic audio waveforms of the video feed to generate a synthetic soundtrack that is synchronized with the video feed,

wherein a generator network is trained during a training phase, comprising:

receiving historical data comprising one or more historic video feeds;

for each historic video feed, dividing the historic video feed into a sequence of video segments;

for each video segment of a historic video feed:

extracting visual features in the video segment;

generating a predicted spectrogram based on the extracted visual features;

comparing a real spectrogram to the predicted spectrogram by a discriminator network to identify a level of similarity; and

upon determining that the level of similarity is above a predetermined threshold, identifying the predicted spectrogram to be a valid representation of the extracted visual features.

2. The computing device of claim 1 , wherein each video segment is a video frame.

3. The computing device of claim 1 , wherein extracting visual features in the video segment comprises identifying one or more topics of the video segment.

4. The computing device of claim 1 , wherein the visual features of the video segment comprise a dominant visual feature and one or more non-dominant visual features.

5. The computing device of claim 4 , wherein a synthetic audio waveform associated with the dominant feature is emphasized, whereas each synthetic audio waveform associated with the one or more non-dominant visual features is de-emphasized.

6. The computing device of claim 1 , wherein the determination is performed substantially in real time.

7. The computing device of claim 1 , wherein generating a predicted spectrogram comprises: providing the extracted visual features into a generator operative to create the predicted spectrogram based on the extracted visual features.

8. The computing device of claim 1 , wherein:

the computing device is a user device;

a camera is coupled to the processor; and

the video feed is received from the camera.

9. A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computing device to carry out a method of adapting a video stream to include a synthetic audio stream, the method comprising:

receiving a video feed;

dividing the video feed into a sequence of video segments;

for each video segment:

extracting visual features of the video segment;

generating a predicted spectrogram based on the extracted visual features; and

generating a synthetic audio waveform from the predicted spectrogram; and

concatenating all synthetic audio waveforms of the video feed to generate a synthetic soundtrack that is synchronized with the video feed,

wherein a generator network is trained during a training phase, comprising:

receiving historical data comprising one or more historic video feeds;

for each historic video feed, dividing the historic video feed into a sequence of video segments;

for each video segment of a historic video feed:

extracting visual features in the video segment;

generating a predicted spectrogram based on the extracted visual features;

comparing a real spectrogram to the predicted spectrogram by a discriminator network to identify a level of similarity; and

upon determining that the level of similarity is above a predetermined threshold, identifying the predicted spectrogram to be a valid representation of the extracted visual features.

10. The non-transitory computer readable storage medium of claim 9 , wherein each video segment is a video frame.

11. The non-transitory computer readable storage medium of claim 9 , wherein extracting visual features in the video segment comprises identifying one or more topics of the video segment.

12. The non-transitory computer readable storage medium of claim 9 , wherein the visual features of the video segment comprise a dominant visual feature and one or more non-dominant visual features.

13. The non-transitory computer readable storage medium of claim 12 , wherein a synthetic audio waveform associated with the dominant feature is emphasized, whereas a synthetic audio waveform associated with the one or more non-dominant visual features is de-emphasized.

14. The computing device of claim 9 , wherein the adaptation of the video stream to include the synthetic audio stream is performed substantially in real time.

15. The non-transitory computer readable storage medium of claim 9 , wherein generating a predicted spectrogram comprises: providing the extracted visual features into a generator operative to create the predicted spectrogram based on the extracted visual features.

16. A computing device comprising:

a processor;

a sound engine configured to perform acts comprising:

dividing the video feed into a sequence of video segments;

for each video segment:

extracting visual features of the video segment;

generating a predicted spectrogram based on the extracted visual features; and

generating a synthetic audio waveform from the predicted spectrogram; and

concatenating all synthetic audio waveforms of the video feed to generate a synthetic soundtrack that is synchronized with the video feed,

wherein generating a predicted spectrogram comprises training a generator network during a training phase, comprising:

receiving historical data comprising one or more historic video feeds;

for each historic video feed, dividing the historic video feed into a sequence of video segments;

for each video segment of a historic video feed;

extracting visual features in the video segment;

generating a predicted spectrogram based on the extracted visual features;

comparing a real spectrogram to the predicted spectrogram by a discriminator network to identify a level of similarity; and

upon determining that the level of similarity is above a predetermined threshold, identifying the predicted spectrogram to be a valid representation of the extracted visual features.

17. The computing device of claim 16 , wherein extracting visual features in the video segment comprises identifying one or more topics of the video segment.

18. The computing device of claim 16 , wherein:

the computing device is a user device;

a camera is coupled to the processor; and

the video feed is received from the camera.