Incremental data processing
View Patent ↗Event logs in a video advertisement insertion network are processed to remove duplicate entries. One or more ad servers are continuously generating new event entries and writing them to a database. The entries are randomized such that generated time contiguous entries are distributed over multiple storage locations, thereby facilitating resource scaling and a uniform use of storage and computing resources. The distributed entries are read from the storage locations using sequential reads of chunks of the entries and processed to remove duplicate entries.
1. A computer-implemented method, comprising:
receiving multiple event records including a first event record and a second event record, wherein the first event record comprises a first event identification and the second event record comprises a second event identification;
generating, for the first event record, a first salt value as a function of the first event identification;
generating, for the second event record, a second salt value as a function of the second event identification;
distributing, based on at least the first salt value and the second salt value, the multiple event records over storage locations;
reading a chunk of event records from the storage locations; and
removing duplicate entries from the chunk based on event identifications to generate de-duplicated event records.
2. The method of claim 1 , further comprising:
reporting the de-duplicated event records to an auditor for verification.
3. The method of claim 1 , wherein the first salt value includes a first hash value of the first event identification and the second salt value includes a second hash value of the second event identification.
4. The method of claim 1 , wherein the storage locations store data using a database that provides contiguous reads and data persistence.
5. The method of claim 1 , wherein the chunk of event records includes 50,000 to 80,000 entries.
6. The method of claim 1 , wherein the multiple event records include information about video advertisements displayed using an online video advertisement exchange.
7. The method of claim 1 , further comprising:
generating unique events in a video advertisement delivery network.
8. The method of claim 7 , wherein generating unique events in a video advertisement delivery network comprises:
receiving a first notification of a first number of events logged at an ad server;
receiving a second notification of a second number of events generated for billing by a billing system;
determining whether the first number and the second number are identical; and
replaying, when the first number and the second number are not identical, a portion of one or more events logged at the ad server until the first number and the second number are identical.
9. The method of claim 8 , further comprising:
operating the billing system to receive a report of events logged from the ad server and perform de-duplication of the events from the report by removing from the report entries that were previously processed by the billing system and included in a calculation of the second number.
10. A computer program product comprising a non-transitory computer-readable program medium having code stored thereon, the code when executed, causing one or more processors to implement a method comprising:
receiving multiple event records including a first event record and a second event record, wherein the first event record comprises at least one of a first event identification, a first event generation time stamp or a first event description field, and the second event record comprises at least one of a second event identification, a second event generation time stamp or a second event description field;
generating, for the first event record, a first salt value as a function of the first event identification;
generating, for the second event record, a second salt value as a function of the second event identification;
distributing, based on at least the first salt value and the second salt value, the multiple event records over storage locations;
reading a chunk of event records from the storage locations; and
removing duplicate entries from the chunk based on event identifications to generate de-duplicated event records.
11. The computer program product of claim 10 , wherein the method further comprises:
reporting the de-duplicated event records to an auditor for verification.
12. The computer program product of claim 10 , wherein the storage locations store data using a database that provides contiguous reads and data persistence.
13. The computer program product of claim 10 , wherein the first salt value includes a first hash value of the first event identification and the second salt value includes a second hash value of the second event identification.
14. The computer program product of claim 10 , wherein the chunk of event records includes 50,000 to 80,000 entries.
15. The computer program product of claim 10 , wherein the multiple event records include information about video advertisements displayed using an online video advertisement exchange.
16. The computer program product of claim 10 , wherein the method further comprises:
receiving a report of events logged; and
perform de-duplication of the events from the report.
17. The computer program product of claim 10 , wherein the method further comprises:
generating unique events in a video advertisement delivery network.
18. The computer program product of claim 17 , wherein generating unique events in a video advertisement delivery network comprises:
receiving a first notification of a first number of events logged at an ad server;
receiving a second notification of a second number of events generated for billing by a billing system;
determining whether the first number and the second number are identical; and
replaying, when the first number and the second number are not identical, a portion of one or more events logged at the ad server until the first number and the second number are identical.
19. A communication system, comprising:
one or more processors that generate event records pertaining to video advertisements served in a communication network and communicate the event records over a transport network; and
a de-duplication mechanism that receives the event records from the transport network and removes duplicate entries from the event records by comparing at least some of the event records with previously received event records within a time window, wherein the de-duplication mechanism includes:
a randomizer module that randomizes the event records to generate randomized event records;
a distribution module that distributes the randomized event records to multiple computer platforms;
a chunking module that reads a subset of the event records on at least one of the multiple computer platforms; and
a comparator module that identifies one or more duplicate entries by comparing an identifier of each entry with one or more previously seen identifiers.
20. The communication system of claim 19 , further including:
an auditor module that compares de-duplicated event records with a report from one or more ad servers for numerical consistency.