Incremental data processing
View Patent ↗Event logs in a video advertisement insertion network are processed to remove duplicate entries. One or more ad servers are continuously generating new event entries and writing them to a database. The entries are randomized such that generated time contiguous entries are distributed over multiple storage locations, thereby facilitating resource scaling and a uniform use of storage and computing resources. The distributed entries are read from the storage locations using sequential reads of chunks of the entries and processed to remove duplicate entries.
1. A computer-implemented method for de-duplicating event records in an event transaction processing system, comprising:
receiving multiple event records, each event record including at least an event generation time stamp, an event identification, and an event description field, wherein each event identification uniquely identifies the corresponding event record;
generating, for each event record, a salt value as a function of the corresponding event identification for that event record;
distributing, canonically based on the salt values, the multiple event records over a number of storage locations;
reading a chunk of event records from the storage locations; and
removing duplicate entries from the chunk based on event identifications to generate de-duplicated event records, wherein each chunk of event records includes entries whose event generation time stamps are continuous with respect to each other and salt values are identical to each other.
2. The method of claim 1 , further comprising:
reporting the de-duplicated event records to an auditor for verification.
3. The method of claim 1 , wherein the salt value includes a hash value of the corresponding event identification.
4. The method of claim 1 , wherein the number of storage locations store data using a database that provides contiguous reads and data persistence.
5. The method of claim 1 , wherein each chunk of event records includes 50,000 to 80,000 entries.
6. The method of claim 1 , wherein the event records include information about video advertisements displayed using an online video advertisement exchange.
7. The method of claim 1 , further comprising:
generating unique events in a video advertisement delivery network.
8. The method of claim 7 , wherein generating unique events in a video advertisement comprises:
receiving a first notification of a first number of events logged at an ad server;
receiving a second notification of a second number of events generated for billing by a billing system;
comparing whether the first number and the second number are identical; and
replaying, when the first number and the second number are not identical, a portion of the events logged at the ad server until the first number and the second number are identical.
9. The method of claim 8 , further comprising:
operating the billing system to receive a report of the events logged from the ad server and perform de-duplication of the events from the report by removing from the report entries that were previously processed by the billing system and included in a calculation of the second number.
10. An apparatus for de-duplicating event logs in an event transaction processing system, comprising:
a module that receives multiple event records, each event record including at least an event generation time stamp, an event identification, and an event description field, wherein each event identification uniquely identifies the corresponding event record;
a module that generates, for each event record, a salt value as a function of the corresponding event identification for that event record; a module that distributes, canonically based on the salt values, the multiple event records over a number of storage locations;
a module that reads a chunk of event records from the storage locations; and
a module that removes duplicate entries from the chunk based on event identifications to generate de-duplicated event records, wherein each chunk of event records includes entries whose event generation time stamps are continuous with respect to each other and salt values are identical to each other.
11. The apparatus of claim 10 , further comprising:
a module that reports the de-duplicated event records to an auditor for verification.
12. The apparatus of claim 10 , wherein the salt value includes a hash value of the corresponding event identification.
13. The apparatus of claim 10 , wherein the number of storage locations store data using a database that provides contiguous reads and data persistence.
14. The apparatus of claim 10 , wherein the event records include information about video advertisements displayed using an online video advertisement exchange.
15. The apparatus of claim 10 , wherein the event identification is a 128 bit global unique identifier generated randomly.
16. A computer program product comprising a non-transitory computer-readable program medium having code stored thereon, the code when executed, causing one or more processors to implement a method of operating a video advertisement exchange system, the method comprising:
receiving multiple event records, each event record including at least an event generation time stamp, an event identification, and an event description field, wherein each event identification uniquely identifies the corresponding event record;
generating, for each event record, a salt value as a function of the corresponding event identification for that event record;
distributing, canonically based on the salt values, the multiple event records over a number of storage locations;
reading a chunk of event records from the storage locations; and
removing duplicate entries from the chunk based on event identifications to generate de-duplicated event records, wherein each chunk of event records includes entries whose event generation time stamps are continuous with respect to each other and salt values are identical to each other.
17. The computer program product of claim 16 , wherein the method further comprises:
reporting the de-duplicated event records to an auditor for verification.
18. The computer program product of claim 16 , wherein the event records include information about video advertisements displayed using an online video advertisement exchange.