Methods, systems, and media for controlling append-only file rewrites

1. A method for controlling append-only-file (AOF) rewrites, comprising:

determining an AOF utilization of a shard of a database;

determining whether the AOF utilization exceeds an AOF utilization threshold;

prior to an AOF rewrite being performed, determining a recovery time that is an estimate of a time to recover the shard from an AOF when performing the AOF rewrite by at least adding write times for each of a plurality of write operations to the shard;

determining whether the recovery time exceeds a recovery time threshold; and

when the AOF utilization is determined to exceed the AOF utilization threshold or when the recovery time is determined to exceed the recovery time threshold, causing the AOF rewrite to be performed.

2. The method of claim 1 , wherein determining the utilization comprises dividing an AOF size for the shard by a maximum AOF size for the shard.

3. The method of claim 1 , wherein determining the recovery time is based on a maximum downtime specified in a service level agreement.

4. The method of claim 1 , wherein determining the recovery time includes:

determining an initial estimated loading time of a snapshot part of the AOF;

for each write operation of a plurality of write operations to the AOF, determining an amount of time required to complete the write operation; and

summing the determined amounts of time for the plurality of write operations with the initial estimate loading time.

5. The method of claim 1 , further comprising:

determining a current total persistent storage utilization;

determining whether the current total persistent storage utilization exceeds a total persistent storage utilization threshold; and

when the current total persistent storage utilization exceeds the total persistent storage utilization threshold, causing an AOF rewrite to be performed.

6. A system for controlling append-only-file (AOF) rewrites, comprising:

at least one hardware processor configured to:

determine an AOF utilization of a shard of a database;

determine whether the AOF utilization exceeds an AOF utilization threshold;

prior to an AOF rewrite being performed, determine a recovery time that is an estimate of a time to recover the shard from an AOF when performing the AOF rewrite by at least adding write times for each of a plurality of write operations to the shard;

determine whether the recovery time exceeds a recovery time threshold; and

when the AOF utilization is determined to exceed the AOF utilization threshold or when the recovery time is determined to exceed the recovery time threshold, cause the AOF rewrite to be performed.

7. The system of claim 6 , wherein the at least one hardware processor determines the utilization by at least dividing an AOF size for the shard by a maximum AOF size for the shard.

8. The system of claim 6 , wherein the at least one hardware processor determines the recovery time based at least on a maximum downtime specified in a service level agreement.

9. The system of claim 6 , wherein the at least one hardware processor determines the recovery time by at least:

determining an initial estimated loading time of a snapshot part of the AOF;

for each write operation of a plurality of write operations to the AOF, determining an amount of time required to complete the write operation; and

summing the determined amounts of time for the plurality of write operations with the initial estimated loading time.

10. The system of claim 6 , wherein the at least one hardware processor is also configured to:

determine a current total persistent storage utilization;

determine whether the current total persistent storage utilization exceeds a total persistent storage utilization threshold; and

when the current total persistent storage utilization exceeds the total persistent storage utilization threshold, cause an AOF rewrite to be performed.

11. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for controlling append-only-file (AOF) rewrites, the method comprising:

determining an AOF utilization of a shard of a database;

determining whether the AOF utilization exceeds an AOF utilization threshold;

prior to an AOF rewrite being performed, determining a recovery time that is an estimate of a time to recover the shard from an AOF when performing the AOF rewrite by at least adding write times for each of a plurality of write operations to the shard;

determining whether the recovery time exceeds a recovery time threshold; and

when the AOF utilization is determined to exceed the AOF utilization threshold or when the recovery time is determined to exceed the recovery time threshold, causing the AOF rewrite to be performed.

12. The non-transitory computer-readable medium of claim 11 , wherein determining the utilization comprises dividing an AOF size for the shard by a maximum AOF size for the shard.

13. The non-transitory computer-readable medium of claim 11 , wherein determining the recovery time is based on a maximum downtime specified in a service level agreement.

14. The non-transitory computer-readable medium of claim 11 , wherein determining the recovery time includes:

determining an initial estimated loading time of a snapshot part of the AOF;

for each write operation of a plurality of write operations to the AOF, determining an amount of time required to complete the write operation; and

summing the determined amounts of time for the plurality of write operations with the initial estimated loading time.

15. The non-transitory computer-readable medium of claim 11 , wherein the method further comprises:

determining a current total persistent storage utilization;

determining whether the current total persistent storage utilization exceeds a total persistent storage utilization threshold; and

when the current total persistent storage utilization exceeds the total persistent storage utilization threshold, causing an AOF rewrite to be performed.