Granted Patent B2

US 12,475,617 · App. 18/517,908 · Granted Nov 18, 2025

Inventors: Hanlin Lu (Los Angeles, CA); Jianbo Yuan (Los Angeles, CA); Xinyu Li (Los Angeles, CA)

Assignee: LEMON INC.

G06T11/206G06N20/00

Abstract

Claims (72)

1 . A method for processing media data, comprising:

obtaining a spectrogram representation for the media data from a spectrogram of the media data, and obtaining a waveform representation for the media data from a waveform of the media data;

generating a fusion representation for the media data based on the spectrogram representation and the waveform representation; and

determining a classification of the media data based on the fusion representation,

wherein the method is implemented by a learning model obtained by:

pretraining the learning model based on unlabeled media data according to self-supervised contrastive learning; and

adjusting the pretrained learning model based on labeled media data.

2 . The method of claim 1 , wherein obtaining the spectrogram representation comprises:

generating a spectrogram map from the spectrogram of the media data;

obtaining a plurality of spectrogram patches from the spectrogram map; and

determining the spectrogram representation based on the plurality of spectrogram patches.

3 . The method of claim 2 , wherein obtaining the waveform representation comprises:

generating a waveform map from the waveform of the media data;

obtaining a plurality of waveform patches from the waveform map, the plurality of waveform patches being aligned to the plurality of the spectrogram patches; and

determining the waveform representation based on the plurality of waveform patches.

4 . The method of claim 1 , wherein generating the fusion representation for the media data based on the spectrogram representation and the waveform representation comprises:

determining an intermedia representation based on a bottleneck representation, the spectrogram representation and the waveform representation according to an attention operation, a dimension of the intermedia representation being equal to a sum of dimensions of the bottleneck representation, the spectrogram representation and the waveform representation; and

extracting the fusion representation from the intermedia representation by removing a portion of the intermedia representation corresponding to the bottleneck representation.

5 . The method of claim 4 , wherein determining the intermedia representation comprises:

determining, based on a concatenation of the spectrogram representation and the bottleneck representation, an intermedia spectrogram representation according to the attention operation;

determining, a concatenation of the waveform representation and the bottleneck representation, an intermedia waveform representation according to the attention operation; and

generating the intermedia representation based on an attention of the intermedia spectrogram representation and the intermedia waveform representation.

6 . The method of claim 1 , wherein pretraining the learning model comprises:

obtaining a first spectrogram representation and a first waveform representation from first media data, the first media data being unlabeled media data; and

updating the learning model according to a contrastive loss that decreases a difference between the first spectrogram representation and the first waveform representation.

7 . The method of claim 6 , wherein pretraining the learning model further comprises:

obtaining a second spectrogram representation and a second waveform representation from second media data, the second media data being unlabeled media data; and

updating the learning model according to a contrastive loss that increases a difference between the first spectrogram representation and the second waveform representation.

8 . The method of claim 6 , wherein pretraining the learning model further comprises:

obtaining reconstructed media data from the learning model based on any of the first spectrogram representation and the first waveform representation; and

updating the learning model based on a reconstruction loss that decreases a difference between the first media data and the reconstructed media data.

9 . The method of claim 1 , wherein adjusting the pretrained learning model comprises:

obtaining an estimated classification from the pretrained learning model based on a third fusion representation of third media data, the third media data being labeled media data; and

updating the pretrained learning model based on a classification loss that decreases a difference between a classification of third media data and the estimated classification.

10 . An electronic device, comprising a computer processor coupled to a computer-readable memory unit, the memory unit comprising instructions that when executed by the computer processor implements a method for processing media data, the method comprising:

obtaining a spectrogram representation for the media data from a spectrogram of the media data, and obtaining a waveform representation for the media data from a waveform of the media data;

generating a fusion representation for the media data based on the spectrogram representation and the waveform representation; and

determining a classification of the media data based on the fusion representation,

wherein the method is implemented by a learning model obtained by:

pretraining the learning model based on unlabeled media data according to self-supervised contrastive learning; and

adjusting the pretrained learning model based on labeled media data.

11 . The device of claim 10 , wherein obtaining the spectrogram representation comprises:

generating a spectrogram map from the spectrogram of the media data;

obtaining a plurality of spectrogram patches from the spectrogram map; and

determining the spectrogram representation based on the plurality of spectrogram patches.

12 . The device of claim 11 , wherein obtaining the waveform representation comprises:

generating a waveform map from the waveform of the media data;

obtaining a plurality of waveform patches from the waveform map, the plurality of waveform patches being aligned to the plurality of the spectrogram patches; and

determining the waveform representation based on the plurality of waveform patches.

13 . The device of claim 10 , wherein generating the fusion representation for the media data based on the spectrogram representation and the waveform representation comprises:

determining an intermedia representation based on a bottleneck representation, the spectrogram representation and the waveform representation according to an attention operation, a dimension of the intermedia representation being equal to a sum of dimensions of the bottleneck representation, the spectrogram representation and the waveform representation; and

extracting the fusion representation from the intermedia representation by removing a portion of the intermedia representation corresponding to the bottleneck representation.

14 . The device of claim 13 , wherein determining the intermedia representation comprises:

determining, based on a concatenation of the spectrogram representation and the bottleneck representation, an intermedia spectrogram representation according to the attention operation;

determining, a concatenation of the waveform representation and the bottleneck representation, an intermedia waveform representation according to the attention operation; and

generating the intermedia representation based on an attention of the intermedia spectrogram representation and the intermedia waveform representation.

15 . The device of claim 10 , wherein pretraining the learning model comprises:

obtaining a first spectrogram representation and a first waveform representation from first media data, the first media data being unlabeled media data; and

updating the learning model according to a contrastive loss that decreases a difference between the first spectrogram representation and the first waveform representation.

16 . The device of claim 15 , wherein pretraining the learning model further comprises:

obtaining a second spectrogram representation and a second waveform representation from second media data, the second media data being unlabeled media data; and

updating the learning model according to a contrastive loss that increases a difference between the first spectrogram representation and the second waveform representation.

17 . The device of claim 15 , wherein pretraining the learning model further comprises:

obtaining reconstructed media data from the learning model based on any of the first spectrogram representation and the first waveform representation; and

updating the learning model based on a reconstruction loss that decreases a difference between the first media data and the reconstructed media data.

18 . A non-transitory computer program product, the non-transitory computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform a method for processing media data, the method comprising:

obtaining a spectrogram representation for the media data from a spectrogram of the media data, and obtaining a waveform representation for the media data from a waveform of the media data;

generating a fusion representation for the media data based on the spectrogram representation and the waveform representation; and

determining a classification of the media data based on the fusion representation,

wherein the method is implemented by a learning model obtained by:

pretraining the learning model based on unlabeled media data according to self-supervised contrastive learning; and

adjusting the pretrained learning model based on labeled media data.

Assignments (2)

Continuity (2)

References Cited (24)