Mechanism for stream processing efficiency using probabilistic model to reduce data redundancy
A method and system of data deduplication for data streams in a multi-tenant system. The method receives, at a data accuracy manager, an event from an activity tracking component, determine whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify the event has not been previously received with a possible false positive response where the event has been previously received, determines whether information for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events, and discards the event in response to determining that the event is recorded in the probabilistic model and in the metric storage.
1. A method of data deduplication for data streams in a multi-tenant system, the method comprising:
receiving, at a data accuracy manager, an event from an activity tracking component;
determining whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify that the event has not been previously received with a possible false positive response where the event has been previously received, wherein determining whether the event is recorded is a check for a primary key for the event;
determining whether the primary key for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events; and
discarding the event in response to determining that the event is recorded in the probabilistic model and the primary key for the event is stored in the metric storage.
2. The method of data deduplication of claim 1 , wherein the probabilistic model is a bloom filter.
3. The method of data deduplication of claim 1 , further comprising:
adding a unique identifier of the event to the probabilistic model in response to failing to find the unique identifier in the probabilistic model.
4. The method of data deduplication of claim 1 , further comprising:
adding the information for the event to the metric storage in response to failing to find a primary key for the event in the probabilistic model.
5. The method of data deduplication of claim 1 , further comprising:
determining whether the probabilistic model exists in response to receiving an event message.
6. The method of data deduplication of claim 5 , further comprising:
initializing the probabilistic model with a unique identifier for the event, in response to determining that the probabilistic model does not exist.
7. The method of data deduplication of claim 1 , wherein the event is generated by an activity tracker.
8. A non-transitory machine-readable storage medium that provides instructions that, if executed by a processor, will cause said processor to perform operations comprising:
receiving, at a data accuracy manager, an event from an activity tracking component;
determining whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify that the event has not been previously received with a possible false positive response where the event has been previously received wherein determining whether the event is recorded is a check for a primary key for the event;
determining whether the primary key for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events; and
discarding the event in response to determining that the event is recorded in the probabilistic model and the primary key for the event is stored in the metric storage.
9. The non-transitory machine-readable storage medium of claim 8 , wherein the probabilistic model is a bloom filter.
10. The non-transitory machine-readable storage medium of claim 8 , the operations further comprising:
adding a unique identifier of the event to the probabilistic model in response to failing to find the unique identifier in the probabilistic model.
11. The non-transitory machine-readable storage medium of claim 8 , the operations further comprising:
adding the information for the event to the metric storage in response to failing to find a primary key for the event in the probabilistic model.
12. The non-transitory machine-readable storage medium of claim 8 , the operations further comprising:
determining whether the probabilistic model exists in response to receiving an event message.
13. The non-transitory machine-readable storage medium of claim 12 , the operations further comprising:
initializing the probabilistic model with a unique identifier for the event, in response to determining that the probabilistic model does not exist.
14. The non-transitory machine-readable storage medium of claim 8 , wherein the event is generated by an activity tracker.
15. A computing device to execute a method of data deduplication for data streams in a multi-tenant system, the computing device comprising:
a non-transitory machine-readable medium having stored therein a data accuracy manager; and
a processor coupled to the non-transitory machine-readable medium, the processor to execute the data accuracy manager to receive an event from an activity tracking component, determine whether the event is recorded in a probabilistic model that tracks previously received events from the activity tracking component, where the probabilistic model can accurately identify that the event has not been previously received with a possible false positive response where the event has been previously received, wherein determining whether the event is recorded is a check for a primary key for the event, to determine whether the primary key for the event is stored in a metric storage, where the metric storage is a database of metrics derived from the previously received events, and to discard the event in response to determining that the event is recorded in the probabilistic model and the primary key for the event is stored in the metric storage.
16. The computing device of claim 15 , wherein the probabilistic model is a bloom filter.
17. The computing device of claim 15 , wherein the data accuracy manager is further to add a unique identifier of the event to the probabilistic model in response to failing to find the unique identifier in the probabilistic model.