CONTEXTUAL ADVERTISING WITH DYNAMICALLY CUSTOMIZED AND/OR GENERATED CREATIVES USING GENERATIVE AI
A system for contextual modification of media content based on multimodal extraction of metadata from the media, wherein the metadata is extracted by processing one or more scenes in the media to extract metadata corresponding to multiple extraction modes, and an embedding model for each extraction mode wherein an aggregated embedding model responsive to the extracted metadata for each mode formulates an aggregated embedding. A process controller may include an embedding extractor responsive to a control input. The control input may specify one or more features appearing in the content defining a media modification opportunity. The embedding extractor may include an embedding model coordinated with the embedding model for one or more of the embedding modes to generate an opportunity embedding in the form of a vector. A vector comparison processor determines the distance between the opportunity embedding and the aggregated embedding, wherein the embeddings are in the form of vectors. The process controller is responsive to the vector comparison processor to generate edit control instructions indicating a modification of the media upon detection of the media modification opportunity. A media content editor is responsive to the edit control instructions to modify the media. The media content editor uses generative AI techniques to modify the media content.
1 . A system for contextual modification of media content based on multimodal extraction of metadata from said media, wherein said metadata is extracted by processing one or more scenes in said media to extract metadata corresponding to multiple extraction modes, and an embedding model for each extraction mode wherein an aggregated embedding model responsive to said extracted metadata for each mode formulates an aggregated embedding comprising:
a process controller including an embedding extractor responsive to a control input wherein said control input specifies one or more features defining a media modification opportunity and wherein said embedding extractor includes an embedding model coordinated with said embedding model for one or more of said embedding modes to generate an opportunity embedding in the form of a vector;
a vector comparison processor for determining the distance between said opportunity embedding and said aggregated embedding to determine a media modification opportunity;
wherein said process controller is responsive to said vector comparison processor to generate edit control instructions indicating a modification of said media upon detection of said media modification opportunity;
a media content editor responsive to said edit control instructions to modify said media and having a modified media output wherein said media content editor uses generative AI techniques to modify said media content.
2 . The system for contextual modification of media based on multimodal extraction of metadata from said media according to claim 1 further comprising a creative library, wherein said creative library stores one or more templates creatives and said edit control instructions specify said generative AI techniques to said template creative retrieved from said creative library application for use by said media content editor.
3 . The system for contextual modification of media content based on multimodal extraction of metadata from said media according to claim 2 wherein said edit control instructions cause said media content editor apply said generative AI techniques based on triggering embeddings.
4 . The system for contextual modification of media based on multimodal extraction of metadata from said media according to claim 3 wherein said edit control instructions include triggering embeddings.
5 . The system for contextual modification of media based on multimodal extraction of metadata from said media according to claim 4 wherein said triggering embeddings are based in part on said aggregated embeddings.
6 . The system for contextual modification of media based on multimodal extraction of metadata from said media according to claim 5 wherein said triggering embeddings are based in part on metadata concerning viewers, and further comprising a viewer database for viewer metadata.
7 . The system for contextual modification of media based on multimodal extraction of metadata from said media according to claim 6 wherein said edit control instructions are dynamically generated based on said viewer database and scene content embeddings in said modified media content.
8 . The system for contextual modification of media based on multimodal extraction of metadata from said media according to claim 7 further comprising a modification selection server responsive to said opportunity to select a modification to apply to said media.
9 . The system for contextual modification of media based on multimodal extraction of metadata from said media according to claim 8 wherein said modification selection server is a competitive bid processor.