System and method for determining effects of multi-channel media sources on multi-channel conversion events
This paper presents a practical method for measuring the impact of multiple marketing events on sales, including marketing events that are not traditionally trackable. The technique infers which of several competing media events are likely to have caused a given conversion. The method is tested using hold-out sets, and also a live media experiment for determining whether the method can accurately predict television-generated web conversions.
1. A method of attributing a plurality of conversion events to a plurality of streamed media events comprising:
automatically aggregating and validating data feeds for each of a plurality of streamed media events;
calculating, by a processor, a first multivariate distribution for each of the plurality of streamed media events, the first multivariate distribution of a streamed media event including variables for time, demographics, and geographic location;
calculating, by the processor, a second multivariate distribution for each of a plurality of voice telephony-based conversion events, the second multivariate distribution of a voice telephony-based conversion event including variables for time, demographics, and geographic location, wherein each voice telephony-based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
calculating, by the processor, a third multivariate distribution for each of a plurality of web based conversion events, the third multivariate distribution of a web based conversion event including variables for time, demographics, and geographic location, wherein each web based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
determining, for each voice telephony-based conversion event of the plurality of voice telephony-based conversion events, a probability that the particular voice telephony-based conversion event was initiated at least partially as a result of an occurrence of a particular streamed media event of the plurality of streamed media events based on a comparison of the first multivariate distribution of the particular streamed media event to the second multivariate distribution of the particular voice telephony-based conversion event;
determining, for each web based conversion event of the plurality of web based conversion events, a probability that the particular web based conversion event was initiated at least partially as a result of an occurrence of a particular streamed media event of the plurality of streamed media events based on a comparison of the first multivariate distribution of the particular streamed media event to the third multivariate distribution of the particular web based conversion event;
automatically attributing, by the processor, one or more of the plurality of voice telephony-based conversion events to a streamed media event of the plurality of streamed media events based on the determined probabilities of the plurality of voice telephony-based conversion events;
automatically attributing, by the processor, one or more of the plurality of web based conversion events to a streamed media event of the plurality of streamed media events based on the determined probabilities of the plurality of web based conversion events;
calculating, by the processor, a correlation between the voice telephony-based conversion events attributed to the streamed media event of the plurality of streamed media events and the web based conversion events attributed to the streamed media event of the plurality of streamed media events;
comparing the calculated correlation to a predetermined threshold;
upon determining that the calculated correlation exceeds a predetermined threshold, attributing, by the processor, the web based conversion events to the streamed media event of the plurality of streamed media events;
generating a report to an advertiser of the attribution of the web based conversion events to the streamed media event; and
allocating an advertising budget according to the reported attribution.
2. The method of claim 1 , wherein each streamed media event is an advertisement using one of the following media: television, radio, Internet, billboard, print, and direct mail, and
wherein each voice telephony-based conversion event and web based conversion event are at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement.
3. The method of claim 1 , wherein determining, for each voice telephony-based conversion event of the plurality of voice telephony-based conversion events, the probability that the particular voice telephony-based conversion event was initiated at least partially as the result of the occurrence of the particular streamed media event of the tuple includes performing one or more of the following tests: KL-disparity, T-tests, multivariate F-tests, correlation, distance, rank tests, and other tests for differences between distributions.
4. The method of claim 1 , wherein the plurality of web based conversion events is at least one of an online purchase of a good that is a subject of the advertisement or an online purchase of the service that is the subject of the advertisement, and
wherein the plurality of voice telephony-based conversion events is at least one of an offline purchase of the good that is the subject of the advertisement or an offline purchase of a service that is the subject of the advertisement.
5. A method comprising:
receiving first data for each of a plurality of streamed media events, wherein each streamed media event includes an advertisement;
automatically aggregating and validating the received first data for the plurality of streamed media events;
receiving second data for each of a plurality of voice telephony-based conversion events, wherein each voice telephony-based conversion event includes at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement, wherein each voice telephony-based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
receiving third data for each of a plurality of web based conversion events, wherein each web based conversion event includes at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement, wherein each web based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
identifying, by the processor, a subset of the plurality of voice telephony-based conversion events and the plurality of web based conversion events as ground truth cases, wherein for a ground truth case, a voice telephony-based conversion event or a web based conversion event is known to be initiated at least partially as a result of an occurrence of a streamed media event;
training, by the processor, a model to predict a probability that a particular voice telephony-based conversion event of the plurality of voice telephony-based conversion events or a particular web based conversion event of the plurality of web based conversion events was initiated at least partially as a result of an occurrence of a particular streamed media event of the plurality of streamed media events based on at least one of a demographic distance between streamed media events and conversion events, viewership of streamed media events, and time difference between streamed media events and voice telephony-based conversion events and web based conversion events;
automatically attributing, by the processor, one or more of the plurality of voice telephony-based conversion events to a streamed media event of the plurality of streamed media events based on the predicted probabilities of the plurality of voice telephony-based conversion events;
automatically attributing, by the processor, one or more of the plurality of web based conversion events to a streamed media event of the plurality of streamed media events based on the predicted probabilities of the plurality of web based conversion events;
calculating, by the processor, a correlation between the voice telephony-based conversion events attributed to the streamed media event of the plurality of streamed media events and the web based conversion events attributed to the streamed media event of the plurality of streamed media events;
comparing the calculated correlation to a predetermined threshold;
upon determining that the calculated correlation exceeds a predetermined threshold, attributing, by the processor, the web based conversion events to the streamed media event of the plurality of streamed media events;
generating a report to an advertiser of the attribution of the web based conversion events to the streamed media event; and
allocating an advertising budget according to the reported attribution.
6. The method of claim 5 , further comprising:
comparing the first data of the particular streamed media event and the second data of the particular voice telephony-based conversion event to the model, and comparing the first data of the particular streamed media event and the third data of the particular web based conversion event to the model.
7. The method of claim 5 , wherein the voice telephony-based conversion event known to be initiated at least partially as the result of the occurrence of the streamed media event if the streamed media event is the only streamed media event preceding the voice telephony-based conversion event by less than a threshold time period and a first geographic location of the streamed media event is less than a threshold distance from a second geographic location of the voice telephony-based conversion event.
8. A non-transitory computer readable medium having instructions that, when executed by a processor, cause the processor to perform operations comprising:
automatically aggregating and validating data feeds for a plurality of streamed media events;
calculating, by a processor, a first multivariate distribution for each of the plurality of streamed media events, the first multivariate distribution of a streamed media event including variables for time, demographics, and geographic location;
calculating, by the processor, a second multivariate distribution for each of a plurality of voice telephony-based conversion events, the second multivariate distribution of a voice telephony-based conversion event including variables for time, demographics, and geographic location, wherein each voice telephony-based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
calculating, by the processor, a third multivariate distribution for each of a plurality of web based conversion events, the third multivariate distribution of a web based conversion event including variables for time, demographics, and geographic location, wherein each web based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
determining, for each voice telephony-based conversion event of the plurality of voice telephony-based conversion events, a probability that the particular voice telephony-based conversion event was initiated at least partially as a result of an occurrence of a particular streamed media event of the plurality of streamed media events based on a comparison of the first multivariate distribution of the particular streamed media event to the second multivariate distribution of the particular voice telephony-based conversion event;
determining, for each web based conversion event of the plurality of web based conversion events, a probability that the particular web based conversion event was initiated at least partially as a result of an occurrence of a particular streamed media event of the plurality of streamed media events based on a comparison of the first multivariate distribution of the particular streamed media event to the third multivariate distribution of the particular web based conversion event;
automatically attributing, by the processor, one or more of the plurality of voice telephony-based conversion events to a streamed media event of the plurality of streamed media events based on the determined probabilities of the plurality of voice telephony-based conversion events;
automatically attributing, by the processor, one or more of the plurality of web based conversion events to a streamed media event of the plurality of streamed media events based on the determined probabilities of the plurality of web based conversion events;
calculating, by the processor, a correlation between the voice telephony-based conversion events attributed to the streamed media event of the plurality of streamed media events and the web based conversion events attributed to the streamed media event of the plurality of streamed media events;
comparing the calculated correlation to a predetermined threshold;
upon determining that the calculated correlation exceeds a predetermined threshold, attributing, by the processor, the web based conversion events to the streamed media event of the plurality of streamed media events;
generating a report to an advertiser of the attribution of the web based conversion events to the streamed media event; and
allocating an advertising budget according to the reported attribution.
9. The non-transitory computer readable medium of claim 8 , wherein each streamed media event is an advertisement using one of the following media: television, radio, Internet, billboard, print, and direct mail, and
wherein each voice telephony-based conversion event and web based conversion event are at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement.
10. The non-transitory computer readable medium of claim 8 , wherein determining, for each voice telephony-based conversion event of the plurality of voice telephony-based conversion events, the probability that the particular voice telephony-based conversion event was initiated at least partially as the result of the occurrence of the particular streamed media event of the tuple includes performing one or more of the following tests: KL-disparity, T-tests, multivariate F-tests, correlation, distance, rank tests, and other tests for differences between distributions.
11. The non-transitory computer readable medium of claim 8 , wherein the plurality of web based conversion events is at least one of an online purchase of a good that is a subject of the advertisement or an online purchase of the service that is the subject of the advertisement, and
wherein the plurality of voice telephony-based conversion events is at least one of an offline purchase of the good that is the subject of the advertisement or an offline purchase of a service that is the subject of the advertisement.
12. A non-transitory computer readable medium having instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving first data for each of a plurality of streamed media events, wherein each streamed media event includes an advertisement;
automatically aggregating and validating the received first data for the plurality of streamed media events;
receiving second data for each of a plurality of voice telephony-based conversion events, wherein each voice telephony-based conversion event includes at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement, wherein each voice telephony-based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
receiving third data for each of a plurality of web based conversion events, wherein each web based conversion event includes at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement, wherein each web based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
identifying, by the processor, a subset of the plurality of voice telephony-based conversion events and the plurality of web based conversion events as ground truth cases, wherein for a ground truth case, a voice telephony-based conversion event or a web based conversion event is known to be initiated at least partially as a result of an occurrence of a streamed media event;
training, by the processor, a model to predict a probability that a particular voice telephony-based conversion event of the plurality of voice telephony-based conversion events or a particular web based conversion event of the plurality of web based conversion events was initiated at least partially as a result of an occurrence of a particular streamed media event of the plurality of streamed media events based on at least one of a demographic distance between streamed media events and conversion events, viewership of streamed media events, and time difference between streamed media events and voice telephony-based conversion events and web based conversion events;
automatically attributing, by the processor, one or more of the plurality of voice telephony-based conversion events to a streamed media event of the plurality of streamed media events based on the predicted probabilities of the plurality of voice telephony-based conversion events;
automatically attributing, by the processor, one or more of the plurality of web based conversion events to a streamed media event of the plurality of streamed media events based on the predicted probabilities of the plurality of web based conversion events;
calculating, by the processor, a correlation between the voice telephony-based conversion events attributed to the streamed media event of the plurality of streamed media events and the web based conversion events attributed to the streamed media event of the plurality of streamed media events;
comparing the calculated correlation to a predetermined threshold;
upon determining that the calculated correlation exceeds a predetermined threshold, attributing, by the processor, the web based conversion events to the streamed media event of the plurality of streamed media events;
generating a report to an advertiser of the attribution of the web based conversion events to the streamed media event; and
allocating an advertising budget according to the reported attribution.
13. The non-transitory computer readable medium of claim 12 , the operations further comprising:
comparing the first data of the particular streamed media event and the second data of the particular voice telephony-based conversion event to the model, and
comparing the first data of the particular streamed media event and the third data of the particular web based conversion event to the model.
14. The non-transitory computer readable medium of claim 12 , wherein the voice telephony-based conversion event known to be initiated at least partially as the result of the occurrence of the streamed media event if the streamed media event is the only streamed media event preceding the voice telephony-based conversion event by less than a threshold time period and a first geographic location of the streamed media event is less than a threshold distance from a second geographic location of the voice telephony-based conversion event.
15. A computing device comprising:
a memory; and
a processor coupled to the memory, the processor to:
receive first data for each of a plurality of streamed media events, wherein each streamed media event includes an advertisement;
automatically aggregate and validate the received first data for the plurality of streamed media events;
receive second data for each of a plurality of voice telephony-based conversion events, wherein each voice telephony-based conversion event includes at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement, wherein each voice telephony-based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
receive third data for each of a plurality of web based conversion events, wherein each web based conversion event includes at least one of a purchase of a good that is a subject of the advertisement and a purchase of a service that is the subject of the advertisement, wherein each web based conversion event is not necessarily known to be associated with one or more of the plurality of streamed media events;
identify a subset of the plurality of voice telephony-based conversion events and the plurality of web based conversion events as ground truth cases, wherein for a ground truth case, a voice telephony-based conversion event or a web based conversion event is known to be initiated at least partially as a result of an occurrence of a streamed media event;
train a model to predict a probability that a particular voice telephony-based conversion event of the plurality of voice telephony-based conversion events or a particular web based conversion event of the plurality of web based conversion events was initiated at least partially as a result of an occurrence of a particular streamed media event of the plurality of streamed media events based on at least one of a demographic distance between streamed media events and conversion events, viewership of streamed media events, and time difference between streamed media events and voice telephony-based conversion events and web based conversion events;
automatically attribute one or more of the plurality of voice telephony-based conversion events to a streamed media event of the plurality of streamed media events based on the predicted probabilities of the plurality of voice telephony-based conversion events;
automatically attribute one or more of the plurality of web based conversion events to a streamed media event of the plurality of streamed media events based on the predicted probabilities of the plurality of web based conversion events;
calculate a correlation between the voice telephony-based conversion events attributed to the streamed media event of the plurality of streamed media events and the web based conversion events attributed to the streamed media event of the plurality of streamed media events
compare the calculated correlation to a predetermined threshold;
upon determining that the calculated correlation exceeds a predetermined threshold, attributing, by the processor, the web based conversion events to the streamed media event of the plurality of streamed media events;
generating a report to an advertiser of the attribution of the web based conversion events to the streamed media event; and
allocating an advertising budget according to the reported attribution.
16. The computing device of claim 15 , wherein the processor is further to:
compare the first data of the particular streamed media event and the second data of the particular voice telephony-based conversion event to the model, and
compare the first data of the particular streamed media event and the third data of the particular web based conversion event to the model.