High-dimensional stratified sampling
In one aspect, a processing device of an information processing system is operative to perform high-dimensional stratified sampling of a database comprising a plurality of records arranged in overlapping sub-groups. For a given record, the processing device determines which of the sub-groups the given record is associated with, and for each of the sub-groups associated with the given record, checks if a sampling rate of the sub-group is less than a specified sampling rate. If the sampling rate of each of the sub-groups is less than the specified sampling rate, the processing device samples the given record, and otherwise does not sample the given record. The determine, check and sample operations are repeated for additional records, and samples resulting from the sample operations are processed to generate information characterizing the database. Other aspects of the invention relate to determining which records to sample through iterative optimization of an objective function that may be based, for example, on a likelihood function of the sampled records.
1. An apparatus comprising:
a processing device comprising a processor having an associated memory;
wherein the processing device is operative:
for a given record, to determine which of a plurality of sub-groups the given record is associated with;
for each of the sub-groups associated with the given record, to check if a sampling rate of the sub-group is less than a specified sampling rate;
if the sampling rate of each of the sub-groups is less than the specified sampling rate, to sample the given record, the given record otherwise not being sampled;
to repeat said determine, check and sample operations for each of a plurality of additional records; and
to maintain for each of the sub-groups a first counter indicating a number of records associated with that sub-group and a second counter indicating a number of times that records from that sub-group have been sampled;
wherein samples resulting from the sample operations are processed to generate information characterizing a database comprising the sub-groups; and
wherein the given record is associated with two or more sub-groups.
2. The apparatus of claim 1 wherein the processing device comprises a controller having a sampling module configured to perform said determine, check and sample operations for the given record and the plurality of additional records.
3. The apparatus of claim 1 wherein the sub-groups comprise overlapping sets of records of the database.
4. The apparatus of claim 1 wherein the processing device is further operative to determine the sampling rate for each sub-group as a function of a value of the first counter maintained for that sub-group and a value of the second counter maintained for that sub-group.
5. The apparatus of claim 4 wherein the processing device is further operative to update at least one of the first and second counters for each sub-group associated with the given record based on whether or not the given record is sampled.
6. The apparatus of claim 1 wherein the database stores N records, each with K attributes, where each attribute takes m k discrete values, 1≦k≦K , and further wherein each sub-group of records takes a particular one of the m k discrete values for each of one or more attributes.
7. The apparatus of claim 1 wherein the given record and the plurality of additional records are obtained and processed sequentially by the processing device.
8. An integrated circuit comprising the apparatus of claim 1 .
9. A processor-implemented method comprising steps of:
for a given record, determining which of a plurality of sub-groups the given record is associated with;
for each of the sub-groups associated with the given record, checking if a sampling rate of the sub-group is less than a specified sampling rate;
if the sampling rate of each of the sub-groups is less than the specified sampling rate, sampling the given record, the given record otherwise not being sampled;
repeating said determining, checking and sampling steps for each of a plurality of additional records;
processing samples resulting from the sampling steps to generate information characterizing a database comprising the sub-groups; and
maintaining for each of the sub-groups a first counter indicating a number of records associated with that sub-group and a second counter indicating a number of times that records from that sub-group have been sampled;
wherein the given record is associated with two or more sub-groups.
10. The method of claim 9 wherein the sub-groups comprise overlapping sets of records of the database.
11. The method of claim 9 further comprising determining the sampling rate for each sub-group as a function of a value of the first counter maintained for that sub-group and a value of the second counter maintained for that sub-group.
12. The method of claim 11 further comprising updating at least one of the first and second counters for each sub-group associated with the given record based on whether or not the given record is sampled.
13. The method of claim 9 wherein the database stores N records, each with K attributes, where each attribute takes m k discrete values, 1≦k≦K , and further wherein each sub-group of records takes a particular one of the m k discrete values for each of one or more attributes.
14. An article of manufacture comprising a non-transitory computer-readable storage medium having embodied therein executable program code that when executed by a processor of a processing device causes the device to perform the steps of:
for a given record, determining which of a plurality of sub-groups the given record is associated with;
for each of the sub-groups associated with the given record, checking if a sampling rate of the sub-group is less than a specified sampling rate;
if the sampling rate of each of the sub-groups is less than the specified sampling rate, sampling the given record, the given record otherwise not being sampled;
repeating said determining, checking and sampling steps for each of a plurality of additional records;
processing samples resulting from the sampling steps to generate information characterizing a database comprising the sub-groups; and
maintaining for each of the sub-groups a first counter indicating a number of records associated with that sub-group and a second counter indicating a number of times that records from that sub-group have been sampled;
wherein the given record is associated with two or more sub-groups.
15. The article of manufacture of claim 14 wherein the sub-groups comprise overlapping sets of records of the database.
16. The article of manufacture of claim 14 wherein the executable program code when executed by the processor of the processing device further cases the device to perform the step of determining the sampling rate for each sub-group as a function of a value of the first counter maintained for that sub-group and a value of the second counter maintained for that sub-group.
17. The article of manufacture of claim 16 wherein the executable program code when executed by the processor of the processing device further cases the device to perform the step of updating at least one of the first and second counters for each sub-group associated with the given record based on whether or not the given record is sampled.
18. The article of manufacture of claim 14 wherein the database stores N records, each with K attributes, where each attribute takes m k discrete values, 1≦k≦K , and further wherein each sub-group of records takes a particular one of the m k discrete values for each of one or more attributes.
19. The article of manufacture of claim 14 wherein the given record and the plurality of additional records are obtained and processed sequentially.
20. The method of claim 9 wherein the given record and the plurality of additional records are obtained and processed sequentially.