Incremental data processing
Event logs in a video advertisement insertion network are processed to remove duplicate entries. One or more ad servers are continuously generating new event entries and writing them to a database. The entries are randomized such that generated time contiguous entries are distributed over multiple storage locations, thereby facilitating resource scaling and a uniform use of storage and computing resources. The distributed entries are read from the storage locations using sequential reads of chunks of the entries and processed to remove duplicate entries.
1. A computer-implemented method, comprising:
receiving multiple event records including a first event record and a second event record;
generating, for the first event record, a first salt value;
generating, for the second event record, a second salt value;
distributing, based on at least the first salt value and the second salt value, the multiple event records over storage locations; and
generating unique events in a video advertisement delivery network associated with the storage locations based on a comparison of a first number of events logged at an ad server with a second number of events generated for billing by a billing system.
2. The method of claim 1 , further comprising:
reporting de-duplicated event records to an auditor for verification.
3. The method of claim 1 , wherein the first salt value includes a first hash value and the second salt value includes a second hash value.
4. The method of claim 1 , wherein the storage locations store data using a database that provides contiguous reads and data persistence.
5. The method of claim 1 , further comprising:
reading a chunk of event records from the storage locations, wherein the chunk of event records includes 50,000 to 80,000 entries.
6. The method of claim 1 , wherein the multiple event records include information about video advertisements displayed using an online video advertisement exchange.
7. The method of claim 1 , further comprising:
reading a chunk of event records from the storage locations; and
removing duplicate entries from the chunk to generate de-duplicated event records.
8. The method of claim 7 , wherein generating unique events in a video advertisement delivery network comprises:
receiving a first notification of the first number of events logged at the ad server;
receiving a second notification of the second number of events generated for billing by the billing system;
determining whether the first number and the second number are identical; and
replaying, when the first number and the second number are not identical, a portion of one or more events logged at the ad server until the first number and the second number are identical.
9. The method of claim 1 , further comprising:
operating the billing system to receive a report of events logged from the ad server and perform de-duplication of the events from the report by removing from report entries that were previously processed by the billing system and included in a calculation of the second number.
10. A computer program product comprising a non-transitory computer-readable program medium having code stored thereon, the code when executed, causing one or more processors to implement a method comprising:
receiving multiple event records including a first event record and a second event record, wherein the first event record comprises at least one of a first event identification, a first event generation time stamp or a first event description field, and the second event record comprises at least one of a second event identification, a second event generation time stamp or a second event description field;
generating, for the first event record, a first salt value;
generating, for the second event record, a second salt value;
distributing, based on at least the first salt value and the second salt value, the multiple event records over storage locations; and
generating unique events in an advertisement delivery network associated with the storage locations based on a comparison of a first number of events logged at an ad server with a second number of events generated for billing by a billing system.
11. The computer program product of claim 10 , wherein the method further comprises:
reporting de-duplicated event records to an auditor for verification.
12. The computer program product of claim 10 , wherein the storage locations store data using a database that provides contiguous reads and data persistence.
13. The computer program product of claim 10 , wherein the first salt value includes a first hash value of the first event identification and the second salt value includes a second hash value of the second event identification.
14. The computer program product of claim 10 , wherein the multiple event records includes 50,000 to 80,000 entries.
15. The computer program product of claim 10 , wherein the multiple event records include information about video advertisements displayed using an online video advertisement exchange.
16. The computer program product of claim 10 , wherein the method further comprises:
receiving a report of events logged; and
perform de-duplication of the events from the report.
17. The computer program product of claim 10 , wherein the method further comprises:
reading a chunk of event records from the storage locations; and
removing duplicate entries from the chunk to generate de-duplicated event records.
18. The computer program product of claim 10 , wherein generating unique events in an advertisement delivery network comprises:
receiving a first notification of the first number of events logged at the ad server;
receiving a second notification of the second number of events generated for billing by the billing system;
determining whether the first number and the second number are identical; and
replaying, when the first number and the second number are not identical, a portion of one or more events logged at the ad server until the first number and the second number are identical.
19. A communication system, comprising:
one or more processors that communicate event records over a transport network; and
a de-duplication mechanism that receives the event records from the transport network, wherein the de-duplication mechanism includes:
a randomizer module that randomizes the event records to generate randomized event records;
a distribution module that distributes the randomized event records to multiple computer platforms;
a chunking module that reads a subset of the event records on at least one of the multiple computer platforms; and
a comparator module that identifies one or more duplicate entries based on unique event identifiers, wherein each unique event identifier comprises a salt value generated as a function of an event identification for a corresponding event record.
20. The communication system of claim 19 , further including:
an auditor module that compares de-duplicated event records with a report from one or more ad servers for numerical consistency.