Optimized full-spectrum order statistics-based cardinality estimation
Systems and methods are disclosed for optimizing full-spectrum cardinality approximations on big data utilizing an optimized order statistics technique. To accomplish the foregoing, a multiset of objects that each corresponds to one of a plurality of objects associated with a resource are obtained. A compound data object is populated at least in part with data that is derived based on generated decimal fraction hash values that correspond to each object in the obtained multiset. The populated compound data object is processed with a full-spectrum arithmetic mean estimation operation that can accurately determine a cardinality estimate for the obtained multiset using less resources and time when compared to traditional techniques. The determination is further made without the need to employ linear counting or bias correction operations on low or high cardinalities. An estimated number of unique objects in the obtained multiset is determined as a result of the processing, and subsequently provided for display or further manipulation.
1. A computer-implemented method comprising:
receiving a request for a number of unique objects in a plurality of objects associated with a resource;
obtaining a multiset of objects, wherein each object of the multiset of objects corresponds to at least one object of the plurality of objects associated with the resource, wherein the multiset of objects comprises at least some of the plurality of objects associated with the resource;
initializing a compound data object having a plurality of portions, wherein each portion of the plurality of portions is unpopulated;
populating at least portions of the compound data object, after the initializing, with data based at least in part on generated decimal fraction hash values that correspond to each object in the multiset of objects,
wherein the populating comprises storing, for a generated hash value, at least one of a first value or a second value into a corresponding portion of the compound data object based at least in part on a comparison of the first value and the second value,
wherein the corresponding portion is based at least in part on a first portion of a corresponding product value, and
wherein the second value is based at least in part on a second portion of the corresponding product value;
processing the compound data object, after the populating, with a single procedure to determine an estimated number of unique objects in the multiset of objects, wherein the single procedure is configured to automatically adjust processing of the compound data object for any actual number of unique objects in the multiset of objects; and
providing the estimated number of unique objects in the multiset of objects.
2. The method of claim 1 , wherein the resource is at least one of a website, a database, a service, or a data store.
3. The method of claim 1 , wherein the first portion of the corresponding product value is an integer portion of the corresponding product value.
4. The method of claim 1 , wherein the second portion of the corresponding product value is a decimal portion of the corresponding product value.
5. The method of claim 1 , wherein the corresponding product value is a product of the generated hash value and a size of the compound data object.
6. The method of claim 1 , the compound data object having a total number of portions based at least in part on an average accuracy requirement.
7. The method of claim 6 , wherein the total number of portions is associated with a size of the compound data object.
8. The method of claim 7 , wherein the populating comprises:
obtaining, for a second generated hash value, a third value that is stored in a second corresponding portion of the compound data object, wherein the second corresponding portion is based at least in part on the second generated hash value and the size of the compound data object,
determining, for the second generated hash value, a fourth value that is based at least in part on the second generated hash value and the size of the compound data object, and
storing, for the second generated hash value, at least one of the third value or the fourth value into the second corresponding portion of the compound data object based at least in part on a comparison of the third value and the fourth value.
9. The method of claim 8 , wherein the second corresponding portion is based at least in part on an integer portion of a corresponding product value, wherein the corresponding product value is a product of the second generated hash value and the size of the compound data object, and wherein the fourth value is based at least in part on a decimal portion of the corresponding product value.
10. The method of claim 1 , wherein the comparison is a minimum value comparison.
11. The method of claim 1 , wherein the compound data object is a feature vector.
12. The method of claim 1 , wherein the single procedure includes at least a balancing factor to facilitate processing of the compound data object for any actual number of unique objects in the multiset of objects.
13. The method of claim 12 , wherein the balancing factor is based at least in part on a number of unpopulated portions of the compound data object.
14. The method of claim 1 , wherein the single procedure is a single complex operation for determining the estimated number of unique objects in the multiset of objects.
15. The method of claim 14 , the processing not including a secondary procedure for determining the estimated number of unique objects in the multiset of objects, the secondary procedure including at least one of a bias correction operation or a linear counting operation.
16. The method of claim 14 , wherein the single complex operation is a full-spectrum cardinality formula.
17. A non-transitory computer storage medium storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
receiving a request for a number of unique objects in a plurality of objects associated with a resource, the request including a time period;
obtaining a multiset of objects, wherein each object of the multiset of objects has a timestamp corresponding to the time period included in the request and corresponds to at least one object of the plurality of objects associated with the resource, wherein the multiset of objects comprises at least some of the plurality of objects associated with the resource;
populating at least portions of a compound data object based at least in part on generated decimal fraction hash values that correspond to each object in the multiset of objects,
wherein the populating comprises storing, for a generated hash value, a value into a corresponding portion of the compound data object based at least in part on a second value,
wherein the corresponding portion is based at least in part on an integer portion of a corresponding product value, and
wherein the second value is based at least in part on a decimal portion of the corresponding product value;
processing the compound data object, after the populating, with a full-spectrum cardinality formula to determine an estimated number of unique objects in the multiset of objects, wherein the estimated number of unique objects is substantially equivalent to an actual number of unique objects in the multiset of objects; and
providing the estimated number of unique objects in the multiset of objects.
18. The non-transitory computer storage medium of claim 17 , the compound data object having a number of portions that is based at least in part on an accuracy requirement.
19. A system comprising:
one or more processors; and
memory comprising instructions that when executed by the one or more processors perform one or more operations comprising:
receiving a request for a number of unique objects in a plurality of objects associated with a resource;
obtaining a multiset of objects, wherein each object of the multiset of objects corresponds to at least one object of the plurality of objects associated with the resource, wherein the multiset of objects comprises at least some of the plurality of objects associated with the resource;
initializing a compound data object having a plurality of portions, wherein each portion of the plurality of portions is unpopulated;
populating at least portions of the compound data object, after the initializing, with data based at least in part on generated decimal fraction hash values that correspond to each object in the multiset of objects, the compound data object having a total number of portions based at least in part on an average accuracy requirement, wherein the total number of portions is associated with a size of the compound data object,
wherein the populating comprises:
obtaining, for a generated hash value, a first value that is stored in a corresponding portion of the compound data object, wherein the corresponding portion is based at least in part on the generated hash value and the size of the compound data object,
determining, for the generated hash value, a second value that is based at least in part on the generated hash value and the size of the compound data object, and
storing, for the generated hash value, at least one of the first value or the second value into the corresponding portion of the compound data object based at least in part on a comparison of the first value and the second value, wherein the corresponding portion is based at least in part on an integer portion of a corresponding product value, wherein the corresponding product value is a product of the generated hash value and the size of the compound data object, and wherein the second value is based at least in part on a decimal portion of the corresponding product value;
processing the compound data object, after the populating, with a single procedure to determine an estimated number of unique objects in the multiset of objects, wherein the single procedure is configured to automatically adjust processing of the compound data object for any actual number of unique objects in the multiset of objects; and
providing the estimated number of unique objects in the multiset of objects.
20. The system of claim 19 , wherein the resource is at least one of a website, a database, a service, or a data store.