Machine learning-based systems and methods for automatically removing protected content from a raw media item

1 . A computer-implemented method for automatically removing protected content from a raw media item, the computer-implemented method comprising:

at a remote media sanitization service being implemented by a distributed network of computers:

receiving, via the distributed network of computers, a subscriber-initiated request to remove protected content from a raw media item;

in response to receiving the subscriber-initiated request:

automatically routing, by the distributed network of computers, the raw media item to an audio recognition service of the remote media sanitization service that comprises one or more trained protected content recognition models;

generating, by the audio recognition service executing the one or more trained protected content recognition models, one or more audio origination labels that indicate an origination source for the protected content associated with one or more temporal segments of the raw media item;

automatically routing, by the distributed network of computers, the raw media item to an audio sanitization service of the remote media sanitization service that comprises a plurality of trained audio separation machine learning models;

generating, by the audio sanitization service executing the plurality of trained audio separation machine learning models, a plurality of sanitized instances of the raw media item that separate the protected content from non-protected portions of the raw media item; and

returning a response to the subscriber-initiated request that uses the one or more audio origination labels and the plurality of sanitized instances of the raw media item to prevent unauthorized electronic distribution of the protected content, wherein returning the response to the subscriber-initiated request includes:

generating, based on executing the one or more trained protected content recognition models, a protected-content segmentation map comprising a plurality of gradient-encoded segments extending from a start to an end of the raw media item, wherein each gradient-encoded segment of the plurality of gradient-encoded segments:

corresponds to a distinct time between the start and the end of the raw media item, and

is assigned a gradient intensity value based on a likelihood that the distinct time within the raw media item includes a presence of one or more protected media assets; and

displaying a graphical user interface that includes the plurality of sanitized instances of the raw media item and the protected-content segmentation map of the raw media item.

2 . The computer-implemented method according to claim 1 , wherein:

the one or more trained protected content recognition models and the plurality of trained audio separation machine learning models are trained using a plurality of synthetic audio composites that simulate real-world combinations of the protected content and non-protected content, and

generating the plurality of synthetic audio composites includes:

(A) obtaining a corpus of protected audio content from a protected-content repository and a corpus of event noises from an event sound repository,

(B) randomly sampling one or more protected audio segments from the corpus of protected audio content and one or more event noise segments from the corpus of event noises,

(C) executing one or more audio augmentation operations on the one or more protected audio segments and the one or more event noise segments to generate one or more augmented protected audio segments and one or more augmented event noise segments,

(D) randomly sampling an impulse-response environment file from an impulse-response environment library comprising a plurality of impulse-response files that each represent acoustic characteristics of a distinct physical environment,

(E) convolving the one or more protected audio segments and the one or more augmented event noise segments with the impulse-response file to generate one or more convolved protected audio segments and one or more convolved event noise segments;

(F) generating a respective synthetic audio composite that includes:

the one or more convolved protected audio segments at one or more first locations in the respective synthetic audio composite and the one or more convolved event noise segments at one or more second locations in the respective synthetic audio composite, and

wherein protected-content classification metadata is stored in association with the respective synthetic audio composite and indicates that the respective synthetic audio composite includes the one or more convolved protected audio segments at the one or more first locations and the one or more convolved event noise segments at the one or more second locations; and

(G) automatically repeating (B)-(F) until the plurality of synthetic audio composites satisfies a training sample diversity threshold.

3 . The computer-implemented method according to claim 2 , wherein using the plurality of synthetic audio composites to train an audio origination machine learning model of the one or more trained protected content recognition models includes:

creating a plurality of audio origination data training samples from the plurality of synthetic audio composites, wherein each of the plurality of audio origination data training samples:

corresponds to a respective synthetic audio composite of the plurality of synthetic audio composites,

includes an input feature comprising the respective synthetic audio composite,

includes a first audio origination source label when the respective synthetic audio composite is simulating a production-source environment, and

includes a second, different from the first, audio origination source label when the respective synthetic audio composite is simulating a live-source environment;

inputting the plurality of audio origination data training samples to the audio origination machine learning model; and

training the audio origination machine learning model to classify the input feature in each of the plurality of audio origination data training samples to one of: the first audio origination source label, and the second audio origination source label.

4 . The computer-implemented method according to claim 2 , wherein using the plurality of synthetic audio composites to train the plurality of trained audio separation machine learning models includes:

creating a plurality of audio separation training data samples from the plurality of synthetic audio composites, wherein each of the plurality of audio separation training data samples:

corresponds to a respective synthetic audio composite of the plurality of synthetic audio composites,

includes an input feature comprising the respective synthetic audio composite, the respective synthetic audio composite including the one or more convolved protected audio segments and the one or more convolved event noise segments, and

includes a target output feature comprising a first separated waveform comprising the one or more convolved protected audio segments and a second separated waveform comprising the one or more convolved event noise segments;

inputting the plurality of audio separation training data samples to the plurality of trained audio separation machine learning models, and

training the plurality of trained audio separation machine learning models to separate the input feature in each of the plurality of audio separation training data samples into the first separated waveform and the second separated waveform.

5 . The computer-implemented method according to claim 1 , wherein:

the raw media item corresponds an audio-video recording comprising synchronized audio data and video data,

the audio data in the audio-video recording includes:

non-protected audio data at least comprising the non-protected portions, including speech audio, crowd noise, and sports-related sounds, and

protected audio data comprising the protected content, wherein the protected content correspond to one or more unlicensed songs.

6 . The computer-implemented method according to claim 5 , wherein generating the plurality of sanitized instances of the raw media item that separate the protected content from the non-protected portions of the raw media item includes:

inputting the raw media item to at least a first trained audio separation machine learning model and a second trained audio separation machine learning model of the plurality of trained audio separation machine learning models, wherein each of the plurality of trained audio separation machine learnings comprise a distinct set of separation and reconstruction weight and bias parameters;

generating, via the distinct set of separation and reconstruction weight and bias parameters associated with the first trained audio separation machine learning model, a first reconstructed audio track of the audio data that reconstructs the non-protected audio data while suppressing spectral components corresponding to the protected audio data;

generating, via the distinct set of separation and reconstruction weight and bias parameters associated with the second trained audio separation machine learning model, a second reconstructed audio track of the audio data that reconstructs the non-protected audio data while suppressing the spectral components corresponding to the protected audio data;

merging the first reconstructed audio track with the video data of the raw media item to generate a first sanitized instance of the plurality of sanitized instances of the raw media item; and

merging the second reconstructed audio track with the video data of the raw media item to generate a second sanitized instance of the plurality of sanitized instances of the raw media item.

7 . The computer-implemented method according to claim 1 , wherein a respective trained protected-content recognition model of the one or more trained protected content recognition models detects the presence of the one or more protected media assets within the raw media item by:

generating, via the respective trained protected-content recognition model, a fingerprint vector of the raw media item that numerically encodes spectral and temporal characteristics of audio data in the raw media item;

constructing, via the respective trained protected-content recognition model, a protected-content search query that includes the fingerprint vector of the raw media item as a search parameter;

executing, via the respective trained protected-content recognition model, the protected-content search query against a protected-content fingerprint repository comprising a plurality of protected-content fingerprint vectors of a corpus of protected-content; and

detecting, via the protected-content recognition model, the one or more temporal segments of the raw media item that include the one or more protected media assets based on executing the protected-content search query.

8 . The computer-implemented method according to claim 1 , wherein:

a protected-content recognition model of the one or more trained protected content recognition models detects that the one or more protected media assets occur across one or more time intervals of the raw media item,

a protected-content localization machine learning model of the one or more trained protected content recognition models detects one or more time-indexed segments of the raw media item indicating where the one or more protected media assets temporally occur within the raw media item,

the one or more temporal segments of the raw media item are detected by an audio origination machine learning model of the one or more trained protected content recognition models and correspond to one or more second time-indexed segments, wherein the one or more second time-indexed segments is a superset of the one or more time-indexed segments, and

assigning the gradient intensity value to each gradient-encoded segment of the plurality of gradient-encoded segments at least includes:

detecting one or more remaining time intervals in the raw media item that do not overlap with the one or more time intervals detected by the protected-content recognition model, the one or more time-indexed segments detected by the protected-content localization machine learning model, and the one or more second time-indexed segments detected by the audio origination machine learning model, and

assigning a pre-defined low gradient intensity value to a subset of the plurality of gradient-encoded segments that correspond to the one or more remaining time intervals.

9 . The computer-implemented method according to claim 8 , wherein assigning the gradient intensity value to each gradient-encoded segment of the plurality of gradient-encoded segments includes:

detecting that the one or more second time-indexed segments comprise a subset of time-indexed segments that are not included in the one or more time-indexed segments detected by the protected-content localization machine learning model,

assigning a pre-defined intermediate gradient intensity value to a subset of the plurality of gradient-encoded segments that correspond to the subset of time-indexed segments,

detecting a common set of time-indexed segments across the one or more time-indexed segments detected by the protected-content localization machine learning model and the one or more second time-indexed segments detected by the audio origination machine learning model, and

assigning a pre-defined high gradient intensity value to a subset of the plurality of gradient-encoded segments that correspond to the common set of time-indexed segments across the one or more time-indexed segments and the one or more second time-indexed segments.

10 . The computer-implemented method according to claim 1 , wherein:

detecting the presence of the one or more protected media assets within the raw media item at least includes detecting that a first protected media asset begins at a first time within the raw media item and that a second protected media asset begins at a second time within the raw media item, and

generating the protected-content segmentation map includes:

detecting that the first time within the raw media item corresponds to a first gradient-encoded segment of the plurality of gradient-encoded segments and that the second time within the raw media item corresponds to a second gradient-encoded segment of the plurality of gradient-encoded segments, and

superimposing a first protected-content marker on the first gradient-encoded segment and a second protected-content marker on the second gradient-encoded segment.

11 . The computer-implemented method according to claim 10 , further comprising:

receiving, via the graphical user interface, a first input selecting the first-protected content marker and a second input selecting the second protected-content marker;

in response to receiving the first input selecting the first-protected content marker, displaying a first protected-content tooltip in association with the first-protected content marker, wherein the first protected-content tooltip includes text indicating a name associated with the first protected media asset and that the first protected media asset begins at the first time within the raw media item; and

in response to receiving the second input selecting the second-protected content marker, displaying a second protected-content tooltip in association with the second-protected content marker, wherein the second-protected content marker includes text indicating a name associated with the second protected media asset and that the second protected media asset begins at the second time within the raw media item.

12 . The computer-implemented method according to claim 1 , wherein the graphical user interface displays the protected-content segmentation map in association with a media playback element and a play-pause control element, separate from native playback controls associated with the media playback element,

the computer-implemented method further comprising:

receiving, via the graphical user interface, an input selecting a respective gradient-encoded segment of the plurality of gradient-encoded segments;

in response to receiving the input:

changing a playback position of the media playback element to correspond to the distinct time associated with the respective gradient-encoded segment;

automatically playing, by the media playback element, the raw media item from the playback position; and

while the media playback element is playing the raw media item, continuously updating a playback progress indicator of the protected-content segmentation map in real time to indicate a current playback position of the raw media item relative to the plurality of gradient-encoded segments;

after receiving the input selecting the respective gradient-encoded segment, receiving a second input selecting the play-pause control element while the media playback element is playing the raw media item; and

pausing the raw media item in the media playback element in response to receiving the second input selecting the play-pause control while the media playback element is playing the raw media item.

13 . The computer-implemented method according to claim 12 , wherein updating the playback progress indicator in real time as the media playback element is playing the raw media item includes:

moving the playback progress indicator along the protected-content segmentation map in synchronization with the current playback position of the raw media item in the media playback element, and

as the playback progress indicator of the protected-content segmentation map is moving in synchronization with the current playback position of the raw media item in the media playback element, dynamically changing an opacity of one or more gradient-encoded segments of the plurality of gradient-encoded segments to visually indicate that the current playback position of the raw media item has progressed beyond the distinct time associated with each of the one or more gradient-encoded segments.

14 . The computer-implemented method according to claim 13 , wherein the graphical user interface includes a first container comprising:

a content readiness indicator that instructs a subscriber to review the raw media item before publication when the presence of the one or more protected media assets is detected in the raw media item,

the protected-content segmentation map,

the play-pause control that, when selected, causes the media playback element to play the raw media item, and

a protected-asset table that comprises one or more rows that each include metadata corresponding to a distinct one of the one or more protected media assets detected within the raw media item.

15 . The computer-implemented method according to claim 14 , wherein the graphical user interface includes:

a second container comprising the media playback element; and

a third container comprising metadata associated with the raw media item, including at least:

a filename associated with the raw media item,

an amount of time required to generate the plurality of sanitized instances of the raw media item,

the subscriber that uploaded the raw media item to the remote media sanitization service,

an upload date associated with the raw media item, and

a duration of the raw media item.

16 . The computer-implemented method according to claim 15 , wherein the graphical user interface includes a fourth container comprising:

a download selectable object that, when selected, causes the plurality of sanitized instances of the raw media item to be downloaded to a target electronic device, and

a plurality of sanitized-version cards that each:

correspond to a distinct sanitized instance of the plurality of sanitized instances of the raw media item,

include a second play-pause control that, when selected, causes the distinct sanitized instance to play in the media playback element,

a distinct download selectable object that, when selected, causes the distinct sanitized instance to be downloaded to the target electronic device,

a copy link selectable object that, when selected, causes a shareable network link associated with the distinct sanitized instance to be copied to a clipboard of the target electronic device, and

a publication readiness indicator that indicates that the distinct sanitized instance is ready to publish when the distinct sanitized instance does not include the one or more protected media assets.

17 . A computer-implemented system comprising:

one or more processors;

a memory;

a computer-readable medium operably coupled to the one or more processors, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the one or more processors, cause a computing device to perform operations comprising:

at a remote media sanitization service being implemented by a distributed network of computers:

receiving, via the distributed network of computers, a subscriber-initiated request to remove protected content from a raw media item;

in response to receiving the subscriber-initiated request:

automatically routing, by the distributed network of computers, the raw media item to an audio recognition service of the remote media sanitization service that comprises one or more trained protected content recognition models;

generating, by the audio recognition service executing the one or more trained protected content recognition models, one or more audio origination labels that indicate an origination source for the protected content associated with one or more temporal segments of the raw media item;

automatically routing, by the distributed network of computers, the raw media item to an audio sanitization service of the remote media sanitization service that comprises a plurality of trained audio separation machine learning models;

generating, by the audio sanitization service executing the plurality of audio separation machine learning models, a plurality of sanitized instances of the raw media item that separate the protected content from non-protected portions of the raw media item; and

returning a response to the subscriber-initiated request that uses the one or more audio origination labels and the plurality of sanitized instances of the raw media item to prevent unauthorized electronic distribution of the protected content, wherein returning the response to the subscriber-initiated request includes:

generating, based on executing the one or more trained protected content recognition models, a protected-content segmentation map comprising a plurality of gradient-encoded segments extending from a start to an end of the raw media item, wherein each gradient-encoded segment of the plurality of gradient-encoded segments:

corresponds to a distinct time between the start and the end of the raw media item, and

is assigned a gradient intensity value based on a likelihood that the distinct time within the raw media item includes a presence of one or more protected media assets; and

displaying a graphical user interface that includes the plurality of sanitized instances of the raw media item and the protected-content segmentation map of the raw media item.

18 . A computer-implemented method for automatically removing protected content from a raw media item, the computer-implemented method comprising:

at a remote media sanitization service being implemented by a distributed network of computers:

receiving, via the distributed network of computers, a subscriber-initiated request to remove protected content from a raw media item;

in response to receiving the subscriber-initiated request:

automatically routing, by the distributed network of computers, the raw media item to an audio recognition service of the remote media sanitization service that comprises one or more trained protected content recognition models;

generating, by the audio recognition service executing the one or more trained protected content recognition models, one or more audio origination labels that indicate an origination source for the protected content associated with one or more temporal segments of the raw media item;

automatically routing, by the distributed network of computers, the raw media item to an audio sanitization service of the remote media sanitization service that comprises a plurality of trained audio separation machine learning models;

generating, by the audio sanitization service executing the plurality of trained audio separation machine learning models, a plurality of sanitized instances of the raw media item that separate the protected content from non-protected portions of the raw media item; and

returning a response to the subscriber-initiated request that uses the one or more audio origination labels and the plurality of sanitized instances of the raw media item to prevent unauthorized electronic distribution of the protected content, wherein:

the one or more trained protected content recognition models and the plurality of trained audio separation machine learning models are trained using a plurality of synthetic audio composites that simulate real-world combinations of the protected content and non-protected content, and

generating the plurality of synthetic audio composites includes:

(A) obtaining a corpus of protected audio content from a protected-content repository and a corpus of event noises from an event sound repository,

(B) randomly sampling one or more protected audio segments from the corpus of protected audio content and one or more event noise segments from the corpus of event noises,

(C) executing one or more audio augmentation operations on the one or more protected audio segments and the one or more event noise segments to generate one or more augmented protected audio segments and one or more augmented event noise segments,

(D) randomly sampling an impulse-response environment file from an impulse-response environment library comprising a plurality of impulse-response files that each represent acoustic characteristics of a distinct physical environment,

(E) convolving the one or more augmented protected audio segments and the one or more augmented event noise segments with the impulse-response environment file to generate one or more convolved protected audio segments and one or more convolved event noise segments;

(F) generating a respective synthetic audio composite that includes:

the one or more convolved protected audio segments at one or more first locations in the respective synthetic audio composite and the one or more convolved event noise segments at one or more second locations in the respective synthetic audio composite, and

wherein protected-content classification metadata is stored in association with the respective synthetic audio composite and indicates that the respective synthetic audio composite includes the one or more convolved protected audio segments at the one or more first locations and the one or more convolved event noise segments at the one or more second locations; and

(G) automatically repeating (B)-(F) until the plurality of synthetic audio composites satisfies a training sample diversity threshold.