System and method of forecasting based on periodical sketching of big data of observations

1. A forecast system, comprising:

one or more processors having access to big data of observations, each observation including a combination of attributes and a time stamp; and

non-transitory computer readable medium storing therein program logic executable by the one or more processors, wherein the one or more processors when executing the program logic are configured to:

for each respective observation of a first plurality of observations, parse the respective observation to determine respective attributes in the respective observation and apply one or more functions to the respective observation one or more times to obtain a respective signature of the respective observation;

for each respective time period of a plurality of time periods, transform a second plurality of observations having time stamps in the respective time period into a respective plurality of sketches of observations in the respective time period, whereby each particular sketch in the respective plurality of sketches is associated with a particular attribute value of a plurality of categorical attribute values and includes signatures of at least some observations among the second plurality of observations, each of the at least some observations having at least one attribute that corresponds to the particular attribute value;

receive a forecast request including forecast constraints, and sparse the forecast request to extract a set of targeted attributes from the forecast constraints;

construct a set expression of the forecast constraints using the set of targeted attributes;

select one or more sample periods from the plurality of time periods and, for each sample period of the one or more selected sample periods:

map the set of targeted attributes to sketches of observations in the sample period to identify a set of sketches corresponding, respectively, to the set of targeted attributes; and

perform a set operation on the set of sketches using the set expression to determine a cardinality corresponding to the sample period; and

generate a forecast result based on one or more cardinalities corresponding to the one or more sample periods.

2. The system of claim 1 , wherein the one or more processors executing the program logic are configured to transform the second plurality of observations having time stamps in the respective time period into the respective plurality of sketches of observations in the respective time period by:

allocating a plurality of sketch bins corresponding, respectively, to the plurality of categorical attribute values, whereby respective ones of the plurality of sketch bins are associated with respective ones of the plurality of categorical attribute values; and

populating the plurality of sketch bins for the respective time period with signatures of observations in the respective time period, such that each populated sketch bin of the plurality of sketch bins is populated with one or more signatures of one or more observations, each of the one or more observations having an attribute that corresponds to the categorical attribute value associated with the populated sketch bin.

3. The system of claim 2 , wherein the plurality of categorical attribute values is associated with a plurality of categories of attributes, each category having one or more categorical attributes, and each categorical attribute having one or more categorical attribute values.

4. The system of claim 1 , wherein the one or more processors executing the program logic are configured to apply a hash function to an observation ID of the respective observation multiple times to generate multiple hashed values of an observation ID as the respective signature.

5. The system of claim 1 , wherein each of the plurality of time periods has a duration equal to that of one day.

6. The system of claim 1 , wherein a duration of the sample period is equal to a duration of each of the plurality of time periods.

7. The system of claim 1 , wherein the second plurality of observations includes at least one million observations.

8. The system of claim 7 , wherein each of the at least one million observations include at least 100 attributes.

9. The system of claim 1 , wherein the set expression includes the set of targeted attributes and one or more relational algebraic operators interrelate the set of targeted attributes.

10. A method of forecasting, comprising:

at one or more computer systems having access to big data of observations, each observation including a combination of attributes and a time stamp:

for each respective observation of a first plurality of observations, parsing the respective observation to determine respective attributes in the respective observations and applying one or more functions to the respective observation one or more times to obtain a respective signature of the respective observation;

for each respective time period of a plurality of time periods, transforming a second plurality of observations having time stamps in the respective time period into a respective plurality of sketches of observations in the respective time period, whereby each particular sketch in the respective plurality of sketches is associated with a particular attribute value of a plurality of categorical attribute values and includes signatures of at least some observations among the second plurality of observations, each of the at least some observations having at least one attribute that corresponds to the particular attribute value;

receiving a forecast request, the forecast request including forecast constraints;

extracting a set of targeted attributes from the forecast constraints;

construct a set expression of the forecast constraints using the set of targeted attributes;

select one or more sample periods from the plurality of time periods;

for each sample period of the one or more selected sample periods:

mapping the set of targeted attributes to sketches of observations in the sample period to identify a set of sketches corresponding, respectively, to the set of targeted attributes; and

performing a set operation on the set of sketches using the set expression to determine a cardinality corresponding to the sample period; and

generating a forecast result based on one or more cardinalities corresponding to the one or more sample periods.

11. The method of claim 10 , wherein transforming the second plurality of observations having time stamps in the respective time period into the respective plurality of sketches of observations in the respective time period includes:

for each of the plurality of time periods, allocating a plurality of sketch bins corresponding, respectively, to the plurality of categorical attribute values, whereby respective ones of the plurality of sketch bins are associated with respective ones of the plurality of categorical attribute values; and

populating the plurality of sketch bins for the respective time period with signatures of observations in the respective time period, such that each populated sketch bin of the plurality of sketch bins is populated with one or more signatures of one or more observations, each of the one or more observations having an attribute that corresponds to the categorical attribute value associated with the populated sketch bin.

12. The method of claim 11 , wherein the plurality of categorical attribute values is associated with a plurality of categories of attributes, each category having one or more categorical attributes, and each categorical attribute having one or more categorical attribute values.

13. The method of claim 10 , wherein applying one or more hash function to the respective observation comprises applying a hash function to an observation ID of the respective observation multiple times to generate multiple hashed values as the respective signature.

14. The method of claim 10 , wherein each of the plurality of time periods has a duration equal to that of one day.

15. The method of claim 10 , wherein a duration of the sample period is equal to a duration of each of the plurality of time periods.

16. The method of claim 10 , wherein the second plurality of observations includes at least one million observations.

17. The method of claim 16 , wherein each of the at least one million observations include at least 100 attributes.

18. The method of claim 10 , wherein the set expression includes the set of targeted attributes and one or more relational algebraic operators interrelate the set of targeted attributes.