Usage record aggregation
In an example embodiment, a solution is provided that aggregates records as they are submitted to a third party (on the write path) rather than performing a real-time aggregation when a request is processed that needs the aggregation (read path). More particularly, in an example embodiment, a caching layer is introduced that avoids having to read all usage events to compute an aggregation when a request is received for aggregated data. The caching layer maintains values for various metrics that require aggregation.
1. A method comprising:
receiving, by one or more processors coupled to non-transitory memory, a first usage record from a first customer system, the first usage record indicating an identification of a first user associated with the first customer system and a quantity of usage of a first unit of measure that occurred during a first predefined time period;
storing, by the one or more processors, the first usage record in a first database;
determining, by the one or more processors, that a usage record aggregation data structure instance exists that uniquely corresponds to both the first user and the first predefined time period, the usage record aggregation data structure instance indicating that usage records of the first user corresponding to the first predefined time period are to be aggregated;
upon determining that the usage record aggregation data structure instance exists, aggregating, by the one or more processors, quantities of usage of the first unit of measure from all usage records, in the first database, that correspond uniquely to both the first user and the first predefined time period;
calculating, by the one or more processors, one or more aggregation metrics based on the aggregated quantities; and
storing, by the one or more processors, the one or more aggregation metrics in a caching layer, along with the identification of the first user and an indication of the first predefined time period.
2. The method of claim 1 , further comprising:
receiving, by the one or more processors, from the first customer system, a request for information;
in response to the request for information, retrieving, by the one or more processors, the one or more aggregation metrics from the first database, based on a matching of the identification of the first user and a matching of the indication of the first predefined time period; and
returning, by the one or more processors, a response to the first customer system based on the retrieved one or more aggregation metrics.
3. The method of claim 1 , further comprising:
receiving, by the one or more processors, a second usage record from the first customer system, the second usage record indicating the identification of the first user and a corrected quantity of usage of the first unit of measure, the corrected quantity of usage being a correction of a quantity of usage of another usage record already stored in the first database;
in response to the receiving of the second usage record, determining, by the one or more processors, whether the one or more aggregation metrics include an aggregation metric requiring re-aggregation when a prior usage record is updated; and
in response to a determination that the one or more aggregation metrics include an aggregation metric requiring reaggregation when a prior usage record is updated, reaggregating, by the one or more processors, quantities of usage of the first unit of measure from all usage records, in the first database, that correspond uniquely to both the first user and the first predefined time period, recalculating the one or more aggregation metrics based on the reaggregated quantities, and storing the recalculated one or more aggregation metrics in the first database, along with the identification of the first user and the indication of the first predefined time period.
4. The method of claim 1 , wherein the one or more aggregation metrics include a sum.
5. The method of claim 1 , wherein the one or more aggregation metrics include a median.
6. The method of claim 1 , wherein the one or more aggregation metrics include a last ever.
7. The method of claim 1 , wherein the one or more aggregation metrics include a maximum.
8. The method of claim 1 , wherein the one or more aggregation metrics include a minimum.
9. A system comprising:
one or more processors; and
a non-transitory memory storing instructions that, when executed by at least one processor among the one or more processors, cause the at least one processor to perform operations comprising:
receiving a first usage record from a first customer system, the first usage record indicating an identification of a first user associated with the first customer system and a quantity of usage of a first unit of measure that occurred during a first predefined time period;
storing the first usage record in a first database;
determining that a usage record aggregation data structure instance exists that uniquely corresponds to both the first user and the first predefined time period, the usage record aggregation data structure instance indicating that usage records of the first user corresponding to the first predefined time period are to be aggregated;
upon determining that the usage record aggregation data structure instance exists, aggregating quantities of usage of the first unit of measure from all usage records, in the first database, that correspond uniquely to both the first user and the first predefined time period;
calculating one or more aggregation metrics based on the aggregated quantities; and
storing the one or more aggregation metrics in a caching layer, along with the identification of the first user and an indication of the first predefined time period.
10. The system of claim 9 , wherein the operations further comprise:
receiving, from the first customer system, a request for information;
in response to the request for information, retrieving the one or more aggregation metrics from the first database, based on a matching of the identification of the first user and a matching of the indication of the first predefined time period; and
returning a response to the first customer system based on the retrieved one or more aggregation metrics.
11. The system of claim 9 , wherein the operations further comprise:
receiving a second usage record from the first customer system, the second usage record indicating the identification of the first user and a corrected quantity of usage of the first unit of measure, the corrected quantity of usage being a correction of a quantity of usage of another usage record already stored in the first database;
in response to the receiving of the second usage record, determining whether the one or more aggregation metrics include an aggregation metric requiring re-aggregation when a prior usage record is updated; and
in response to a determination that the one or more aggregation metrics include an aggregation metric requiring reaggregation when a prior usage record is updated, reaggregating quantities of usage of the first unit of measure from all usage records, in the first database, that correspond uniquely to both the first user and the first predefined time period, recalculating the one or more aggregation metrics based on the reaggregated quantities, and storing the recalculated one or more aggregation metrics in the first database, along with the identification of the first user and the indication of the first predefined time period.
12. The system of claim 9 , wherein the one or more aggregation metrics include a sum.
13. The system of claim 9 , wherein the one or more aggregation metrics include a median.
14. The system of claim 9 , wherein the one or more aggregation metrics include a mean.
15. The system of claim 9 , wherein the one or more aggregation metrics include a maximum.
16. The system of claim 9 , wherein the one or more aggregation metrics include a minimum.
17. A non-transitory machine-readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a first usage record from a first customer system, the first usage record indicating an identification of a first user associated with the first customer system and a quantity of usage of a first unit of measure that occurred during a first predefined time period;
storing the first usage record in a first database;
determining that a usage record aggregation data structure instance exists that uniquely corresponds to both the first user and the first predefined time period, the usage record aggregation data structure instance indicating that usage records of the first user corresponding to the first predefined time period are to be aggregated;
upon determining that the usage record aggregation data structure instance exists, aggregating quantities of usage of the first unit of measure from all usage records, in the first database, that correspond uniquely to both the first user and the first predefined time period;
calculating one or more aggregation metrics based on the aggregated quantities; and
storing the one or more aggregation metrics in a caching layer, along with the identification of the first user and an indication of the first predefined time period.
18. The non-transitory machine-readable medium of claim 17 , wherein the operations further comprise:
receiving, from the first customer system, a request for information;
in response to the request for information, retrieving the one or more aggregation metrics from the first database, based on a matching of the identification of the first user and a matching of the indication of the first predefined time period; and
returning a response to the first customer system based on the retrieved one or more aggregation metrics.
19. The non-transitory machine-readable medium of claim 17 , wherein the operations further comprise:
receiving a second usage record from the first customer system, the second usage record indicating the identification of the first user and a corrected quantity of usage of the first unit of measure, the corrected quantity of usage being a correction of a quantity of usage of another usage record already stored in the first database;
in response to the receiving of the second usage record, determining whether the one or more aggregation metrics include an aggregation metric requiring re-aggregation when a prior usage record is updated; and
in response to a determination that the one or more aggregation metrics include an aggregation metric requiring reaggregation when a prior usage record is updated, reaggregating quantities of usage of the first unit of measure from all usage records, in the first database, that correspond uniquely to both the first user and the first predefined time period, recalculating the one or more aggregation metrics based on the reaggregated quantities, and storing the recalculated one or more aggregation metrics in the first database, along with the identification of the first user and the indication of the first predefined time period.
20. The non-transitory machine-readable medium of claim 17 , wherein the one or more aggregation metrics include a sum.