System and method for counting and storing events as scalar quantities in skewed counting environments
View Patent ↗A computer system that includes at least one processor, a first storage resource, and a second storage resource maintains a count of key events. The system determines that the count is not significant and, in response, stores data associated with the key in the second storage resource. The system subsequently can determine when the count has become significant and, in response, stores the data associated with the key in the first storage resource.
1. A method performed by a computer system comprising at least one processor, the method comprising:
defining a plurality of keys, each of the plurality of keys being associated with a key event that is defined at least in part by an occurrence of a user activity;
monitoring activity of a population of users;
determining, from monitoring activity of the population of users, a count of the key event for each key;
determining a significance level of each key in the plurality of keys based at least in part on the count for the key event of each key in the plurality of keys, wherein at least some keys are determined to be significant based on the count for the associated key event exceeding a threshold, and wherein a majority of the plurality of keys are determined to be insignificant based on the count for the associated key event being less than the threshold;
storing a data set for each key based on the determined significance level of that key, each data set including the key and the count for the key event of that key;
wherein storing the data set includes storing the data set of keys that are determined to be significant in a first memory, and the data set of keys that are determined to be insignificant in a second memory, the first memory being more lossless than the second memory;
monitoring the count of the key events for at least the keys that are determined to be insignificant in order to determine when the count for the key event of a given key exceeds the threshold; and
in response to determining that the count for the key event of the given key exceeds the threshold, storing the data set for the given key in the first memory, and setting the count for the key event to be within a predetermined margin of accuracy.
2. The method of claim 1 , wherein determining the significance level is based at least in part on exceeding a predetermined value for a given time period.
3. The method of claim 1 , wherein the second memory uses lossy compression.
4. The method of claim 3 , wherein the first memory uses lossless compression.
5. The method of claim 1 , wherein the key event comprises one of i) a click, ii) an impression, iii) a conversion event, iv) a unique visit, or v) a combination thereof.
6. The method of claim 1 , wherein determining comprises determining a significance level of each key in the plurality of keys comprises determining that the data associated with the key is not stored in the first memory.
7. The method of claim 1 , wherein determining the significance level includes determining that the key has an intermediate level of significance in between an insignificant level and a most significant level.
8. The method of claim 7 , wherein storing data includes using a memory or resource that is associated with the intermediate level of significance, and not associated with the insignificant level or the most significant level.
9. A computer system, comprising:
one or more processors; and
a memory storing instructions that, when executed by the at least one processor, cause the one or more processors to:
define a plurality of keys, each of the plurality of keys being associated with a key event that is defined at least in part by an occurrence of a user activity;
monitor activity of a population of users;
determine, from monitoring activity of the population of users, a count of the key event for each key;
determine a significance level of each key in the plurality of keys based at least in part on the count for the key event of each key in the plurality of keys, wherein at least some keys are determined to be significant based on the count for the associated key event exceeding a threshold, and wherein a majority of the plurality of keys are determined to be insignificant based on the count for the associated key event being less than the threshold;
store a data set for each key based on the determined significance level of that key, each data set including the key and the count for the key event of that key;
wherein storing the data set includes storing the data set of keys that are determined to be significant in a first memory, and the data set of keys that are determined to be insignificant in a second memory, the first memory being more lossless than the second memory;
monitor the count of the key events for at least the keys that are determined to be insignificant in order to determine when the count for the key event of a given key exceeds the threshold; and
in response to determining that the count for the key event of the given key exceeds the threshold, storing the data set for the given key in the first memory, and setting the count for the key event to be within a predetermined margin of accuracy.
10. The computer system of claim 9 , wherein the instructions to determine whether the count is significant comprise instructions to determine whether the count exceeds a predetermined value.
11. The computer system of claim 9 , wherein:
the instructions to store the data associated with the key in the second memory comprise instructions to store the data using lossy compression; and
the instructions to store the data associated with the key in the first memory comprise instructions to store the data using lossless compression.
12. The computer system of claim 9 , further comprising instructions for determining that the key has an intermediate level of significance in between an insignificant level and a most significant level.
13. The computer system of claim 12 , wherein storing data includes using a storage format or memory that is associated with the intermediate level of significance, and not associated with the insignificant level or the most significant level.
14. A non-transitory computer-readable medium that stores instructions, which when executed by one or more processors, cause the one or more processors to perform operations comprising:
defining a plurality of keys, each of the plurality of keys being associated with a key event that is defined at least in part by an occurrence of a user activity;
monitoring activity of a population of users;
determining, from monitoring activity of the population of users, a count of the key event for each key;
determining a significance level of each key in the plurality of keys based at least in part on the count for the key event of each key in the plurality of keys, wherein at least some keys are determined to be significant based on the count for the associated key event exceeding a threshold, and wherein a majority of the plurality of keys are determined to be insignificant based on the count for the associated key event being less than the threshold;
storing a data set for each key based on the determined significance level of that key, each data set including the key and the count for the key event of that key;
wherein storing the data set includes storing the data set of keys that are determined to be significant in a first memory, and the data set of keys that are determined to be insignificant in a second memory, the first memory being more lossless than the second memory;
monitoring the count of the key events for at least the keys that are determined to be insignificant in order to determine when the count for the key event of a given key exceeds the threshold; and
in response to determining that the count for the key event of the given key exceeds the threshold, storing the data set for the given key in the first memory, and setting the count for the key event to be within a predetermined margin of accuracy.
15. The non-transitory computer-readable medium of claim 14 , wherein determining the significance level is based at least in part on exceeding a predetermined value for a given time period.
16. The non-transitory computer-readable medium of claim 14 , wherein the second memory uses lossy compression.
17. The non-transitory computer-readable medium of claim 16 , wherein the first memory uses lossless compression.
18. The non-transitory computer-readable medium of claim 14 , wherein the key event comprises one of i) a click, ii) an impression, iii) a conversion event, iv) a unique visit, or v) a combination thereof.
19. The non-transitory computer-readable medium of claim 14 , wherein determining a significance level of each key in the plurality of keys comprises determining that the data associated with the key is not stored in the first memory.
20. The non-transitory computer-readable medium of claim 14 , wherein determining the significance level includes determining that the key has an intermediate level of significance in between an insignificant level and a most significant level.