Geo, segment, uniques distributed computing system

1. A computer-implemented method of operating a digital media advertisement system, comprising:

receiving a plurality of files from a plurality of ad servers, each file including a plurality of line items, wherein each line item corresponds to an ad delivery instance;

processing, utilizing a first code base, the plurality of files through a pipeline of multiple processing stages, the processing comprising:

controlling intervening load balancers positioned before each of the multiple processing stages to select files from the plurality of files and provide the selected files to a corresponding processing stage from the multiple processing stages;

controlling storage of the selected files received by the corresponding processing stage to a distributed computing cloud;

controlling storage of output data written by the corresponding processing stage based on the selected files to the distributed computing cloud; and

controlling sharding of processing tasks included in the corresponding processing stage based on partitioning the selected files into at least two separate logical groups; and

cross-checking the processing of the plurality of files through the pipeline using a second code base that is different than the first code base utilized to process the plurality of files through the pipeline to determine whether a result of processing the plurality of files using the first code base matches a result of processing the plurality of files using the second code base;

wherein each processing stage of the multiple processing stages receives processing results of a preceding processing stage.

2. The method of claim 1 , further comprising:

generating results based on processing the plurality of files at a headroom interval before expiration of a target time interval after the plurality of files are received from the plurality of ad servers; and

using a measure of the headroom interval to control resources allocated to the pipeline of multiple processing stages.

3. The method of claim 1 , wherein cross-checking the processing of the plurality of files comprises:

checking a result of the processing of the plurality of files by receiving a list identifying the plurality of files and checking whether all files were used in generating the result of the processing of the plurality of files.

4. The method of claim 1 , further comprising:

transferring a plurality of data files from a plurality of geographically distributed advertisement servers to a first storage resource in the distributed computing cloud;

providing a script-based program to the distributed computing cloud;

executing, using resources from the distributed computing cloud, the script-based program to perform analysis of the plurality of data files; and

storing results of the analysis on a second storage resource, wherein the results include at least one operational parameter of the digital media advertisement system.

5. The method of claim 4 , wherein the operational parameter comprises geographic information related to the digital media advertisement system.

6. The method of claim 4 , wherein the operational parameter comprises segment data for consumers of the digital media advertisement system.

7. The method of claim 4 , wherein the operational parameter comprises unique instances of advertisement deliveries in the digital media advertisement system.

8. The method of claim 1 , wherein processing the plurality of files through a pipeline of multiple processing stages comprises:

determining whether all the plurality of ad servers have checked in; and

responsive to determining all the plurality of ad servers having checked in, releasing messages waiting in one of the multiple processing stages to a next processing stage.

9. The method of claim 1 , wherein the ad delivery instance comprises a video advertisement delivery instance.

10. The method of claim 1 , wherein processing the plurality of files through a pipeline of multiple processing stages comprises:

releasing messages waiting in one of the multiple processing stages to a next processing stage when a time period threshold expires.

11. A digital media advertisement system, comprising:

an ad data infrastructure mechanism configured to receive a plurality of files from a plurality of ad servers, each file including a plurality of line items, wherein each line item corresponds to an ad delivery instance; and

a processor configured to:

process, utilizing a first code base, the plurality of files through a pipeline of multiple processing stages, wherein to process the plurality of files the processor is configured to:

control intervening load balancers positioned before each of the multiple processing stages to select files from the plurality of files and provide the selected files to a corresponding processing stage from the multiple processing stages;

control storage of the selected files received by the corresponding processing stage to a cloud service;

control storage of output data written by the corresponding processing stage based on the selected files to the cloud service; and

control sharding of processing tasks included in the corresponding processing stage based on partitioning the selected files into at least two separate logical groups; and

cross-check the processing of the plurality of files through the pipeline using a second code base that is different than the first code base utilized to process the plurality of files through the pipeline to determine whether a result of processing the plurality of files using the first code base matches a result of processing the plurality of files using the second code base;

wherein each processing stage of the multiple processing stages receives processing results of a preceding processing stage.

12. The system of claim 11 , wherein the processor is further configured to:

generate results based on processing the plurality of files at a headroom interval before expiration of a target time interval after the plurality of files are received from the plurality of ad servers; and

use a measure of the headroom interval to control resources allocated to the pipeline of multiple processing stages.

13. The system of claim 11 , wherein to cross-check the processing of the plurality of files through the pipeline, the processor is further configured to:

check a result of the processing of the plurality of files by receiving a list identifying the plurality of files and checking whether all files were used in generating the result of the processing of the plurality of files.

14. The system of claim 11 , wherein the ad delivery instance comprises demographic or geographic information of a viewer of a delivered ad.

15. A computer program product comprising a computer-readable non-transitory medium having code stored thereon, the code, when executed by one or more processors, causing the one or more processors to implement a method of operating a digital media advertisement system, comprising:

receiving a plurality of files from a plurality of ad servers, each file including a plurality of line items, wherein each line item corresponds to an ad delivery instance;

processing, utilizing a first code base, the plurality of files through a pipeline of multiple processing stages, the processing comprising:

controlling intervening load balancers positioned before each of the multiple processing stages to select files from the plurality of files and provide the selected files to a corresponding processing stage from the multiple processing stages;

controlling storage of the selected files received by the corresponding processing stage to a cloud service;

controlling storage of output data written by the corresponding processing stage based on the selected files to the cloud service; and

controlling sharding of processing tasks included in the corresponding processing stage based on partitioning the selected files into at least two separate logical groups; and

cross-checking the processing of the plurality of files through the pipeline using a second code base that is different than the first code base utilized to process the plurality of files through the pipeline to determine whether a result of processing the plurality of files using the first code base matches a result of processing the plurality of files using the second code base;

wherein each processing stage of the multiple processing stages receives processing results of a preceding processing stage.

16. The computer program product of claim 15 , wherein the method further comprises:

generating results based on processing the plurality of files at a headroom interval before expiration of a target time interval after the plurality of files are received from the plurality of ad servers; and

using a measure of the headroom interval to control resources allocated to the pipeline of multiple processing stages.

17. The computer program product of claim 15 , wherein the method further comprises:

checking a result of the processing of the plurality of files by receiving a list identifying the plurality of files and checking whether all files were used in generating the result of the processing of the plurality of files.