Methods, systems, and media for controlling append-only file rewrites

1. A method for controlling append-only-file (AOF) rewrites, comprising:

prior to an AOF rewrite being performed, determining using a hardware processor a recovery time that is an estimate of a time to recover a shard from an AOF when performing the AOF rewrite by at least adding write times for each of a plurality of write operations to the shard;

determining whether the recovery time exceeds a recovery time threshold; and

when the recovery time is determined to exceed the recovery time threshold, causing the AOF rewrite to be performed.

2. The method of claim 1 , wherein the recovery time threshold is based on a maximum downtime specified in a service level agreement.

3. The method of claim 1 , wherein determining the recovery time includes:

determining an initial estimated loading time of a snapshot part of the AOF;

for each write operation of a plurality of write operations to the AOF, determining an amount of time required to complete the write operation; and

summing the determined amounts of time for the plurality of write operations with the initial estimate loading time.

4. The method of claim 3 , wherein the initial estimated loading time is based on a number of keys in the shard.

5. The method of claim 3 , wherein determining the amount of time required to complete the write operation includes measuring an execution time of a write operation to the AOF.

6. The method of claim 1 , further comprising:

determining a current total persistent storage utilization;

determining whether the current total persistent storage utilization exceeds a total persistent storage utilization threshold; and

when the current total persistent storage utilization exceeds the total persistent storage utilization threshold, causing an AOF rewrite to be performed.

7. A system for controlling append-only-file (AOF) rewrites, comprising:

memory; and

at least one hardware processor configured to at least:

determine whether the AOF utilization exceeds an AOF utilization threshold;

prior to an AOF rewrite being performed, determine a recovery time that is an estimate of a time to recover a shard from an AOF when performing the AOF rewrite by at least adding write times for each of a plurality of write operations to the shard;

determine whether the recovery time exceeds a recovery time threshold; and

when the recovery time is determined to exceed the recovery time threshold, cause the AOF rewrite to be performed.

8. The system of claim 7 , wherein the recovery time threshold is based on a maximum downtime specified in a service level agreement.

9. The system of claim 7 , wherein the at least one hardware processor determines the recovery time by at least:

determining an initial estimated loading time of a snapshot part of the AOF;

for each write operation of a plurality of write operations to the AOF, determining an amount of time required to complete the write operation; and

summing the determined amounts of time for the plurality of write operations with the initial estimated loading time.

10. The system of claim 9 , wherein the initial estimated loading time is based on a number of keys in the shard.

11. The system of claim 9 , wherein determining the amount of time required to complete the write operation includes measuring an execution time of a write operation to the AOF.

12. The system of claim 7 , wherein the at least one hardware processor is also configured to:

determine a current total persistent storage utilization;

determine whether the current total persistent storage utilization exceeds a total persistent storage utilization threshold; and

when the current total persistent storage utilization exceeds the total persistent storage utilization threshold, cause an AOF rewrite to be performed.

13. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for controlling append-only-file (AOF) rewrites, the method comprising:

prior to an AOF rewrite being performed, determining a recovery time that is an estimate of a time to recover a shard from an AOF when performing the AOF rewrite by at least adding write times for each of a plurality of write operations to the shard;

determining whether the recovery time exceeds a recovery time threshold; and

when the recovery time is determined to exceed the recovery time threshold, causing the AOF rewrite to be performed.

14. The non-transitory computer-readable medium of claim 13 , wherein the recovery time threshold is based on a maximum downtime specified in a service level agreement.

15. The non-transitory computer-readable medium of claim 13 , wherein determining the recovery time includes:

determining an initial estimated loading time of a snapshot part of the AOF;

for each write operation of a plurality of write operations to the AOF, determining an amount of time required to complete the write operation; and

summing the determined amounts of time for the plurality of write operations with the initial estimated loading time.

16. The non-transitory computer-readable medium of claim 15 , wherein the initial estimated loading time is based on a number of keys in the shard.

17. The non-transitory computer-readable medium of claim 15 , wherein determining the amount of time required to complete the write operation includes measuring an execution time of a write operation to the AOF.

18. The non-transitory computer-readable medium of claim 13 , wherein the method further comprises:

determining a current total persistent storage utilization;

determining whether the current total persistent storage utilization exceeds a total persistent storage utilization threshold; and

when the current total persistent storage utilization exceeds the total persistent storage utilization threshold, causing an AOF rewrite to be performed.