Managing a distributed system processing a publisher's streaming data
A distributed system processing a publisher's streaming data. The distributed system comprises multiple workers and publisher data stores, each publisher data store dedicated to a worker and a publisher. A sampling ratio (the fraction of data items for storage in the publisher's data store) is selected by a publisher data store's worker based on historical information. At least two workers select different sampling ratios. Data items representing an interaction between an entity and the publisher are received. Each data item is assigned to a worker for processing. A hash function is applied to the data item's identifier, resulting in a key value falling within the hash function's range. The scope of the publisher's data store is equal to the hash function's range multiplied by the sampling ratio of the publisher's data store. A data item with a key value within the scope of the publisher's data store is stored therein.
1. A method of managing a distributed system processing streaming data received over a network from a data source, the distributed system comprising a plurality of data stores and a plurality of workers, wherein each of the plurality of data stores is dedicated to the data source, and wherein each of the plurality of workers is dedicated to a respective data store, the method comprising:
selecting, by each of the workers, based on a criteria, a sampling ratio, the sampling ratio defining a fraction of data items selected for storage in the respective data store, wherein a first worker selects a first sampling ratio and a second worker selects a second sampling ratio, and wherein the first and second sampling ratios are different;
responsive to receiving, by the distributed system, streaming data comprising data items from the data source:
assigning, by the distributed system, data items from the streaming data to the workers; and
storing, by each of the assigned workers, based on the respective sampling ratios of each of the assigned workers, data items from the assigned data items in the respective data store;
responsive to receiving, by the distributed system, a query request, wherein the query request comprises a constraint:
distributing, by the distributed system, the query request to each of the workers;
processing, by each of the workers, the query request, wherein the processing comprises:
determining a tally of data items stored in the respective data store meeting the constraint of the query; and
estimating a count of data items assigned to the worker meeting the constraint of the query by dividing the tally by the respective sampling ratio of the respective worker;
estimating, by the distributed system, a total of data items received by the distributed system from the data source meeting the constraint of the query by totaling the estimated counts of data items from each of the workers; and
providing, by the distributed system, the estimated total of data items received by the distributed system from the data source meeting the criteria.
2. The method of claim 1 , wherein the criteria is based on a historical sampling ratio of the respective data store.
3. The method of claim 1 , wherein the criteria is based on a historical received data item volume of the respective data store.
4. The method of claim 1 , wherein the assigning is based on the workload of each of the workers.
5. The method of claim 1 , wherein the assigning is based on the location of each of the workers and on the location where the data item was received.
6. The method of claim 1 , wherein the data items each comprise an identifier of an entity, and wherein the storing further comprises:
computing, by each worker, a key value by applying a hash function to each assigned data item's identifier, the hash function having a range and the key value falling within the hash function's range;
computing, by each worker, the respective data store scope, wherein the respective data store scope is equal to the hash function's range multiplied by the data store's selected sampling ratio;
identifying, by each worker, key assigned data items from the assigned data items where the key value of the data item falls within the corresponding data store's scope; and
storing, by each worker, data items from the key assigned data items, based on the sampling ratio.
7. The method of claim 6 , wherein each data item represents an interaction between the entity and a publisher.
8. The method of claim 1 , wherein receiving the query request further comprises receiving the query request over the network from the data source and wherein providing the estimated total of data items further comprises providing over the network the estimated total of data items to the data source.
9. The method of claim 1 , wherein selecting the sampling ratio further comprises:
responsive to determining, by the first worker, that the number of data items stored in the respective data store using the first sampling ratio exceeds an upper limit, selecting, by the first worker, a third sampling ratio, wherein the third sampling ratio is smaller than the first sampling ratio.
10. A system comprising:
a processor; and
a computer readable storage medium storing processor-executable computer program instructions, the computer program instructions comprising instructions for managing a distributed system processing streaming data received over a network from a data source, the distributed system comprising a plurality of data stores and a plurality of workers, wherein each of the plurality of data stores is dedicated to the data source, and wherein each of the plurality of workers is dedicated to a respective data store, the method comprising:
selecting, by each of the workers, based on a criteria, a sampling ratio, the sampling ratio defining a fraction of data items selected for storage in the respective data store, wherein a first worker selects a first sampling ratio and a second worker selects a second sampling ratio, and wherein the first and second sampling ratios are different;
responsive to receiving, by the distributed system, streaming data comprising data items from the data source:
assigning, by the distributed system, data items from the streaming data to the workers; and
storing, by each of the assigned workers, based on the respective sampling ratios of each of the assigned workers, data items from the assigned data items in the respective data store;
responsive to receiving, by the distributed system, a query request, wherein the query request comprises a constraint:
distributing, by the distributed system, the query request to each of the workers;
processing, by each of the workers, the query request, wherein the processing comprises:
determining a tally of data items stored in the respective data store meeting the constraint of the query; and
estimating a count of data items assigned to the worker meeting the constraint of the query by dividing the tally by the respective sampling ratio of the respective worker;
estimating, by the distributed system, a total of data items received by the distributed system from the data source meeting the constraint of the query by totaling the estimated counts of data items from each of the workers; and
providing, by the distributed system, the estimated total of data items received by the distributed system from the data source meeting the criteria.
11. The system of claim 10 , wherein the criteria is based on a historical sampling ratio of the respective data store.
12. The system of claim 10 , wherein the criteria is based on a historical received data item volume of the respective data store.
13. The system of claim 10 , wherein the assigning is based on the workload of each of the workers.
14. The system of claim 10 , wherein the assigning is based on the location of each of the workers and on the location where the data item was received.
15. The system of claim 10 , wherein the data items each comprise an identifier of an entity, and wherein the storing further comprises:
computing, by each worker, a key value by applying a hash function to each assigned data item's identifier, the hash function having a range and the key value falling within the hash function's range;
computing, by each worker, the respective data store scope, wherein the respective data store scope is equal to the hash function's range multiplied by the data store's selected sampling ratio;
identifying, by each worker, key assigned data items from the assigned data items where the key value of the data item falls within the corresponding data store's scope; and
storing, by each worker, data items from the key assigned data items, based on the sampling ratio.
16. The system of claim 15 , wherein each data item represents an interaction between the entity and a publisher.
17. The system of claim 10 , wherein receiving the query request further comprises receiving over the network the query request from the data source and wherein providing the estimated total of data items further comprises providing over the network the estimated total of data items to the data source.
18. The method of claim 10 , wherein selecting the sampling ratio further comprises:
responsive to determining, by the first worker, that the number of data items stored in the respective data store using the first sampling ratio exceeds an upper limit, selecting, by the first worker, a third sampling ratio, wherein the third sampling ratio is smaller than the first sampling ratio.
19. A non-transitory computer-readable medium with computer executable instructions executed by a processor to perform a method of managing a distributed system processing streaming data received over a network from a data source, the distributed system comprising a plurality of data stores and a plurality of workers, wherein each of the plurality of data stores is dedicated to the data source, and wherein each of the plurality of workers is dedicated to a respective data store, the method comprising:
selecting, by each of the workers, based on a criteria, a sampling ratio, the sampling ratio defining a fraction of data items selected for storage in the respective data store, wherein a first worker selects a first sampling ratio and a second worker selects a second sampling ratio, and wherein the first and second sampling ratios are different;
responsive to receiving, by the distributed system, streaming data comprising data items from the data source:
assigning, by the distributed system, data items from the streaming data to the workers; and
storing, by each of the assigned workers, based on the respective sampling ratios of each of the assigned workers, data items from the assigned data items in the respective data store;
responsive to receiving, by the distributed system, a query request, wherein the query request comprises a constraint:
distributing, by the distributed system, the query request to each of the workers;
processing, by each of the workers, the query request, wherein the processing comprises:
determining a tally of data items stored in the respective data store meeting the constraint of the query; and
estimating a count of data items assigned to the worker meeting the constraint of the query by dividing the tally by the respective sampling ratio of the respective worker;
estimating, by the distributed system, a total of data items received by the distributed system from the data source meeting the constraint of the query by totaling the estimated counts of data items from each of the workers; and
providing, by the distributed system, the estimated total of data items received by the distributed system from the data source meeting the criteria.
20. The method of claim 19 , wherein receiving the query request further comprises receiving the query request from the data source and wherein providing the estimated total of data items further comprises providing the estimated total of data items to the data source.