Media attribution verification
In general, the disclosure describes techniques for obtaining, by a computing system, a content item and a purported source for the content item, wherein the content item may include multimodal data. The techniques may further include generating, by the computing system, a plurality of modality feature vectors representative of the multimodal data, wherein each of the generated modality feature vectors has a different, corresponding modality feature. The techniques may further include mapping, by the computing system, the generated modality feature vectors based on a statistical distribution associated with the purported source. The techniques may further include determining, by the computing system, a score based on the mapping. The techniques may further include outputting, by the computing system and based on the score, an indication of whether the content item originated from the purported source.
1 . A method comprising:
obtaining, by a computing system, a content item and a purported media source for the content item, the content item comprising multimodal data;
generating, by the computing system, a plurality of modality feature vectors representative of the multimodal data, wherein each of the plurality of generated modality feature vectors has a different, corresponding modality feature;
mapping, by the computing system, the plurality of generated modality feature vectors based on a statistical distribution associated with the purported media source;
determining, by the computing system, a score based on the mapping; and
outputting, by the computing system and based on the score, an indication of whether the content item originated from the purported media source.
2 . The method of claim 1 , wherein mapping the plurality of generated modality feature vectors based on the statistical distribution associated with the purported media source further comprises:
creating, by the computing system, a source profile for the purported media source based on training data samples; and
mapping, by the computing system, using the source profile, the plurality of generated modality feature vectors based on the statistical distribution associated with the purported media source.
3 . The method of claim 1 , further comprising:
outputting, by the computing system, an indication of whether the content item originated from other media sources if the content item did not originate from the purported media source.
4 . The method of claim 1 , wherein generating the plurality of modality feature vectors representative of the multimodal data further comprises:
providing the content item to one or more neural networks trained to encode items of a given content type;
extracting multimodal data for a plurality of features from the content item, each feature of the plurality of features corresponding to a modality feature of the content item; and
for each feature of the plurality of features, encoding multimodal data associated with the feature into a corresponding modality feature vector.
5 . The method of claim 1 , wherein each of the plurality of generated modality feature vectors corresponds to a modality feature comprising one of a title of the content item, a body of the content item, one or more images of the content item, one or more captions of the content item, audio of the content item, or video of the content item, and wherein mapping the plurality of generated modality feature vectors based on the statistical distribution associated with the purported media source comprises generating a feature mapping for each modality feature vector.
6 . The method of claim 5 , wherein determining the score based on the mapping comprises:
determining a feature score for each feature mapping; and
determining the score by fusing each of the feature scores.
7 . The method of claim 1 , wherein generating the plurality of modality feature vectors comprises fusing the plurality of generated modality feature vectors into a concatenated modality feature vector, and wherein mapping the plurality of generated modality feature vectors comprises mapping the concatenated modality feature vector based on the statistical distribution associated with the purported media source.
8 . The method of claim 1 , wherein determining the score based on the mapping comprises:
generating one or more raw score values using probabilistic linear discriminant analysis; and
generating the indication of whether the content item originated from the purported media source by calibrating the one or more raw score values using multiclass logistic regression.
9 . A computing system comprising:
an input device configured to receive a content item and a purported media source for the content item, the content item comprising multimodal data;
processing circuitry and memory configured to execute a machine learning system,
wherein the machine learning system is configured to:
generate a plurality of modality feature vectors representative of the multimodal data of the content item, wherein each of the plurality of generated modality feature vectors has a different, corresponding modality feature;
map the plurality of generated modality feature vectors based on a statistical distribution associated with the purported media source;
determine a score based on the mapping; and
output, based on the score, an indication of whether the content item originated from the purported media source.
10 . The computing system of claim 9 , wherein to map the plurality of generated modality feature vectors based on the statistical distribution associated with the purported media source, the machine learning system is further configured to:
create a source profile for the purported media source based on training data samples; and
map the plurality of generated modality feature vectors based on the statistical distribution associated with the purported media source using the source profile.
11 . The computing system of claim 9 , wherein the machine learning system is further configured to:
output an indication of whether the content item originated from other media sources if the content item did not originate from the purported media source.
12 . The computing system of claim 9 , wherein to generate the plurality of modality feature vectors representative of the multimodal data, the machine learning system is further configured to:
provide the content item to one or more neural networks trained to encode items of a given content type;
extract multimodal data for a plurality of features from the content item, a feature of the plurality of features corresponding to a modality feature of the content item; and
for each feature of the plurality of features, encoding multimodal data associated with the feature into a corresponding modality feature vector.
13 . The computing system of claim 9 , wherein each of the plurality of generated modality feature vectors correspond to a modality feature comprising a title of the content item, a body of the content item, one or more images of the content item, one or more captions of the content item, audio of the content item, or video of the content item, and wherein to determine the score based on the mapping, the machine learning system is configured to:
generate a feature mapping for each modality feature vector;
determine a feature score for each feature mapping; and
determine the score by fusing each of the feature scores.
14 . The computing system of claim 9 , wherein to generate the plurality of modality feature vectors, the machine learning system is further configured to:
fuse the plurality of generated modality feature vectors into a concatenated modality feature vector; and
map the concatenated modality feature vector based on the statistical distribution associated with the purported media source.
15 . The computing system of claim 9 , wherein to determine the score based on the mapping, the machine learning system is configured to:
generate one or more raw score values using probabilistic linear discriminant analysis; and
generate the indication of whether the content item originated from the purported media source by calibrating the one or more raw score values using multiclass logistic regression.
16 . The computing system of claim 9 , wherein the indication of whether the content item originated from the purported media source comprises one of a probability, ratio, or loglikelihood ratio.
17 . Non-transitory computer-readable storage media comprising machine readable instructions for configuring processing circuitry to:
obtain a content item and a purported media source for the content item, the content item comprising multimodal data;
generate a plurality of modality feature vectors representative of the multimodal data, wherein each of the plurality of generated modality feature vectors has a different, corresponding modality feature;
map the plurality of generated modality feature vectors based on a statistical distribution associated with the purported media source;
determine a score based on the mapping; and
output, based on the score, an indication of whether the content item originated from the purported media source.
18 . The non-transitory computer-readable storage media of claim 17 , wherein each of the plurality of generated modality feature vectors correspond to a modality feature comprising a title of the content item, a body of the content item, one or more images of the content item, one or more captions of the content item, audio of the content item, or video of the content item, and wherein to determine the score based on the mapping, the processing circuitry is configured to:
generate a feature mapping for each modality feature vector;
determine a feature score for each feature mapping; and
determine the score by fusing each of the feature scores.
19 . The method of claim 1 , further comprising generating the statistical distribution according to a plurality of training content items corresponding to the purported media source.
20 . The method of claim 1 , wherein the indication of whether the content item originated from the purported media source comprises an indication of whether the content item was published by the purported media source.