Profile-guided adaptive data extraction job scheduler for data backup service
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for scheduling jobs for a data backup service using backup process history and backup target entity profiles.
1. A system comprising:
one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
extracting, by first workers of a backup service, entities (a) of a set of entities from an information service and (b) for backing up of the entities in sequential backup cycles;
scheduling jobs in each backup cycle to extract the entities, wherein each job is performed by one of a plurality of workers of the backup service, the jobs being assigned to workers from the plurality of workers in order of current expected runtimes of the extraction of the entities being extracted, from longest to shortest expected runtime; and
for each entity and for each backup cycle after an initial backup cycle calculating a current expected runtime for the entity from:
a learning factor alpha,
a damping factor lambda,
a download time DT, being a runtime to download the entity in an immediately preceding backup cycle,
a size ratio Ratio, being a ratio of (i) a current size of the entity obtained from the information service to (ii) a size of the entity as downloaded in the immediately preceding backup cycle, and
a previous expected runtime ER, being an expected runtime calculated for the entity for the immediately preceding backup cycle,
the current expected runtime being calculated according to current expected runtime=Ratio×(ER+alpha×(DT+lambda×ER−ER)).
2. The system of claim 1 , wherein:
the set of entities includes entities that are tables of a database.
3. The system of claim 1 , wherein:
alpha is determined for each backup cycle after the initial backup cycle as 1/n, wherein n is a number incremented with each backup cycle; and
lambda is a value between zero and one, inclusive.
4. The system of claim 3 , wherein:
an initial value of alpha is one with n equal to one and n is incremented by one with each backup cycle.
5. The system of claim 1 , wherein:
the backup service provides an interface for a user of the information service to determine a value for lambda.
6. The system of claim 1 , wherein a first number of the workers from the plurality of workers is less than a second number of the entities.
7. The system of claim 1 , wherein the operations comprise:
determining a degree to which data stored at the information service is sporadic; and
computing lambda using the degree to which the data stored at the information service is sporadic.
8. The system of claim 1 , wherein, for each entity and for each backup cycle, after the initial backup cycle, calculating the current expected runtime for the entity uses the lambda that represents a degree to which the current expected runtime for a corresponding entity depends on the previous expected runtime ER.
9. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
extracting, by first workers of a backup service, entities (a) of a set of entities from an information service and (b) for backing up of the entities in sequential backup cycles;
scheduling jobs in each backup cycle to extract the entities, wherein each job is performed by one of a plurality of workers of the backup service, the jobs being assigned to workers from the plurality of workers in order of current expected runtimes of the extraction of the entities being extracted, from longest to shortest expected runtime; and
for each entity and for each backup cycle after an initial backup cycle calculating a current expected runtime for the entity from:
a learning factor alpha,
a damping factor lambda,
a download time DT, being a runtime to download the entity in an immediately preceding backup cycle,
a size ratio Ratio, being a ratio of (i) a current size of the entity obtained from the information service to (ii) a size of the entity as downloaded in the immediately preceding backup cycle, and
a previous expected runtime ER, being an expected runtime calculated for the entity for the immediately preceding backup cycle,
the current expected runtime being calculated according to current expected runtime=Ratio×(ER+alpha×(DT+lambda×ER−ER)).
10. The one or more computer-readable storage media of claim 9 , wherein:
the set of entities includes entities that are tables of a database.
11. The one or more computer-readable storage media of claim 9 , wherein:
alpha is determined for each backup cycle after the initial backup cycle as 1/n, wherein n is a number incremented with each backup cycle; and
lambda is a value between zero and one, inclusive.
12. The one or more computer-readable storage media of claim 9 , wherein:
an initial value of alpha is one with n equal to one and n is incremented by one with each backup cycle.
13. The one or more computer-readable storage media of claim 9 , wherein:
the backup service provides an interface for a user of the information service to determine a value for lambda.
14. The one or more computer-readable storage media of claim 6 , wherein the operations comprise:
determining a degree to which data stored at the information service is sporadic; and
computing lambda using the degree to which the data stored at the information service is sporadic.
15. A method comprising:
extracting, by first workers of a backup service, entities (a) of a set of entities from an information service and (b) for backing up of the entities in sequential backup cycles;
scheduling jobs in each backup cycle to extract the entities, wherein each job is performed by one of a plurality of workers of the backup service, the jobs being assigned to workers from the plurality of workers in order of current expected runtimes of the extraction of the entities being extracted, from longest to shortest expected runtime; and
for each entity and for each backup cycle after an initial backup cycle calculating a current expected runtime for the entity from:
a learning factor alpha,
a damping factor lambda,
a download time DT, being a runtime to download the entity in an immediately preceding backup cycle,
a size ratio Ratio, being a ratio of (i) a current size of the entity obtained from the information service to (ii) a size of the entity as downloaded in the immediately preceding backup cycle, and
a previous expected runtime ER, being an expected runtime calculated for the entity for the immediately preceding backup cycle,
the current expected runtime being calculated according to current expected runtime=Ratio×(ER+alpha×(DT+lambda×ER−ER)).
16. The method of claim 15 , wherein:
the set of entities includes entities that are tables of a database.
17. The method of claim 15 , wherein:
alpha is determined for each backup cycle after the initial backup cycle as 1/n, wherein n is a number incremented with each backup cycle; and
lambda is a value between zero and one, inclusive.
18. The method of claim 15 , wherein:
an initial value of alpha is one with n equal to one and n is incremented by one with each backup cycle.
19. The method of claim 15 , wherein:
the backup service provides an interface for a user of the information service to determine a value for lambda.
20. The method of claim 15 , comprising:
determining a degree to which data stored at the information service is sporadic; and
computing lambda using the degree to which the data stored at the information service is sporadic.