Tunable, efficient monitoring of capacity usage in distributed storage systems

1. A method, comprising:

obtaining a set of expiration times associated with usage of a resource;

selecting, by a computer system, a first limit to a number of time slots for use in calculating usage statistics for the resource based on a memory efficiency associated with calculating the usage statistics for the resource;

creating, based on the set of expiration times, a set of time slots after a current time;

populating, by the computer system, the set of time slots with the set of expiration times, up to the first limit in the number of time slots;

when a time slot in the set of time slots includes the current time, using a subset of the expiration times in the time slot to update one or more usage statistics for the resource; and

outputting the one or more usage statistics for use in managing the usage of the resource.

2. The method of claim 1 , further comprising:

identifying a forecast period spanning the set of time slots; and

triggering, at an end of the forecast period, a recalculation of usage statistics for the resource from usage records associated with the resource.

3. The method of claim 2 , further comprising:

when a second limit to the number of time slots is exceeded by creating a new time slot within the forecast period:

deleting a latest time slot prior to the end of the forecast period; and

shortening the forecast period to exclude a future time interval spanned by the deleted latest time slot.

4. The method of claim 3 , wherein the second limit is equal to or higher than the first limit.

5. The method of claim 2 , further comprising:

when a second limit to the number of time slots is not exceeded by creating a new time slot after the forecast period, extending the forecast period to include a future time interval spanned by the new time slot.

6. The method of claim 2 , wherein the usage records comprise:

a create record; and

a delete record.

7. The method of claim 1 , wherein using the subset of the expiration times in the time slot to update the one or more usage statistics for the resource comprises:

subtracting usage of the resource associated with the subset of the expiration times from the one or more usage statistics.

8. The method of claim 1 , wherein outputting the one or more usage statistics comprises:

aggregating the one or more usage statistics with one or more additional usage statistics for a set of users to obtain one or more total usage statistics for the users.

9. The method of claim 1 , wherein the resource comprises a storage resource in a distributed storage system.

10. The method of claim 9 , wherein the one or more usage statistics comprise a capacity usage of the distributed storage system.

11. The method of claim 9 , wherein the expiration times are assigned to binary objects in the distributed storage system.

12. The method of claim 1 , wherein the one or more usage statistics are updated during at least one of:

a beginning of the time slot;

a middle of the time slot; and

an end of the time slot.

13. An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

obtain a set of expiration times associated with usage of a resource;

select a first limit to a number of time slots for use in calculating usage statistics for the resource based on a memory efficiency associated with calculating the usage statistics for the resource;

create, based on the set of expiration times, a set of time slots after a current time;

populate the set of time slots with the set of expiration times, up to the first limit in the number of time slots;

when a time slot in the set of time slots includes the current time, use a subset of the expiration times in the time slot to update one or more usage statistics for the resource; and

output the one or more usage statistics for use in managing the usage of the resource.

14. The apparatus of claim 13 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

identify a forecast period spanning the set of time slots; and

trigger, at an end of the forecast period, a recalculation of usage statistics for the resource from usage records associated with the resource.

15. The apparatus of claim 14 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

when a second limit to the number of time slots is exceeded by creating a new time slot within the forecast period:

delete a latest time slot prior to the end of the forecast period; and

shorten the forecast period to exclude a future time interval spanned by the latest time slot.

16. The apparatus of claim 15 , wherein the second limit is equal to or higher than the first limit.

17. The apparatus of claim 14 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

when a second limit to the number of time slots is not exceeded by creating a new time slot after the forecast period, extend the forecast period to include a future time interval spanned by the new time slot.

18. The apparatus of claim 13 , wherein using the subset of the expiration times in the time slot to update the one or more usage statistics for the resource comprises:

subtracting usage of the resource associated with the subset of the expiration times from the one or more usage statistics.

19. The apparatus of claim 13 , wherein outputting the one or more usage statistics comprises:

aggregating the one or more usage statistics with one or more additional usage statistics for a set of users to obtain one or more total usage statistics for the users.

20. A system, comprising:

a set of partitions distributed across a set of storage nodes; and

a lead partition in the set of partitions, wherein the partition comprises a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to:

obtain a set of expiration times associated with usage of the storage nodes;

select a first limit to a number of time slots for use in calculating usage statistics for the resource based on a memory efficiency associated with calculating the usage statistics for the resource;

create, based on the set of expiration times, a set of time slots after a current time;

populate the set of time slots with the set of expiration times up to the first limit in the number of time slots;

when a time slot in the set of time slots includes the current time, use a subset of the expiration times in the time slot to update one or more usage statistics; and

output the one or more usage statistics for use in managing the usage of the storage nodes.