Efficient resource utilization in data centers

1. A method comprising:

identifying high-availability jobs and low-availability jobs that demand usage of resources of a distributed system;

determining a first quota of the resources available to low-availability jobs as a quantity of the resources available during normal operations;

determining a second quota of the resources available to high-availability jobs as a quantity of the resources available during normal operations minus a quantity of the resources lost due to a tolerated event;

executing the jobs on the resources of the distributed system; and

constraining a total usage of the resources by both the high-availability jobs and the low-availability jobs to the quantity of the resources available during normal operations; and

leaving jobs assigned to the lost resources without reassigning the jobs to reassigning resources; and

reconstructing any unavailable data associated with the lost resources.

2. The method of claim 1 , wherein the resources comprise data processing devices, networking systems, power systems, or cooling systems.

3. The method of claim 2 , further comprising migrating or re-executing jobs assigned to resources lost due to the tolerated event to remaining resources.

4. The method of claim 1 , wherein the resources comprise non-transitory memory devices.

reconstructing any unavailable data associated with the lost resources.

5. The method of claim 1 , further comprising determining the second quota of the resources available to high-availability jobs as:

Q 2 =R N −R L −R I

wherein Q 2 is the second quota, R N is the quantity of the resources available during normal operations, R L is the quantity of the resources lost due to a tolerated event, and R I is an increased quantity of the remaining resources needed due to the tolerated event.

6. The method of claim 1 , further comprising determining the second quota of the resource available to high-availability jobs as:

Q

2

=

(

R

N

-

R

L

)

(

R

N

+

x

*

R

L

)

wherein Q 2 is the second quota, R N is the quantity of the resource available during normal operations, R L is the quantity of the resource lost due to a tolerated event, and x is a resource multiplier corresponding to the tolerated event.

7. The method of claim 6 , wherein the resource comprises non-transitory memory devices.

8. The method of claim 1 , further comprising limiting a sum of the first quota and the second quota to a maximum quota.

9. The method of claim 1 , further comprising:

monitoring a usage of the resources by the high-availability jobs; and

when a high-availability job exceeds a threshold usage, downgrading the high-availability job to a low-availability job.

10. The method of claim 1 , further comprising suspending or ending at least some of the low-availability jobs for the period of time before the tolerated event.

11. The method of claim 1 , further comprising determining the quantity of the resources lost due to a tolerated event based on an assignment of the jobs to particular resources and a system hierarchy of the distributed system, the system hierarchy comprising system domains, each having an active state or an inactive state.

12. The method of claim 1 , wherein the system hierarchy comprises system levels comprising:

a first system level corresponding to host machines of data processing devices, non-transitory memory devices, or network interface controllers, each host machine having a system domain;

a second system level corresponding to power deliverers, communication deliverers, or cooling deliverers of racks housing the host machines, each power deliverer, communication deliverer, or cooling deliverer of the rack having a system domain; and

a third system level corresponding to power deliverers, communication deliverers, or cooling deliverers of cells having associated racks, each power deliverer, communication deliverer, or cooling deliverer of the cell having a system domain; and

a fourth system level corresponding to a distribution center module of the cells, each distribution center module having a system domain.

13. The system of claim 1 , wherein the resources comprises non-transitory memory devices.

14. A system comprising:

resources of a distributed system; and

a computer processor in communication with the resources, the computer processor:

identifying high-availability jobs and low-availability jobs that demand usage of the resources;

determining a first quota of the resources available to low-availability jobs as a quantity of the resources available during normal operations;

determining a second quota of the resources available to high-availability jobs as a quantity of the resources available during normal operations minus a quantity of the resources lost due to a tolerated event;

executing the jobs on the resources of the distributed system; and

wherein the computer processor:

leaves jobs assigned to the lost resources without reassigning the jobs to remaining resources; and

reconstructs any unavailable data associated with the lost resources.

15. The system of claim 14 , wherein the resources comprise data processing devices, networking systems, power systems, or cooling systems.

16. The system of claim 15 , wherein the computer processor migrates or re-executes jobs assigned to resources lost due to the tolerated event to remaining resources.

17. The system of claim 14 , wherein the computer processor determines the second quota of the resources available to high-availability jobs as:

Q 2 =R N −R L −R I

wherein Q 2 is the second quota, R N is the quantity of the resources available during normal operations, R L is the quantity of the resources lost due to a tolerated event, and R I is an increased quantity of the remaining resources needed due to the tolerated event.

18. The system of claim 14 , wherein the computer processor determines the second quota of the resource available to high-availability jobs as:

Q

2

=

(

R

N

-

R

L

)

(

R

N

+

x

*

R

L

)

wherein Q 2 is the second quota, R N is the quantity of the resource available during normal operations, R L is the quantity of the resource lost due to a tolerated event, and x is a resource multiplier corresponding to the tolerated event.

19. The system of claim 18 , wherein the resources comprise non-transitory memory devices.

20. The system of claim 14 , wherein the computer processor limits a sum of the first quota and the second quota to a maximum quota.

21. The system of claim 14 , wherein the computer processor:

monitors a usage of the resources by the high-availability jobs; and

when a high-availability job exceeds a threshold usage, downgrades the high-availability job to a low-availability job.

22. The system of claim 14 , wherein the computer processor suspends or ends at least some of the low-availability jobs for the period of time before the tolerated event.

23. The system of claim 14 , wherein the computer processor determines the quantity of the resources lost due to a tolerated event based on an assignment of the jobs to particular resources and a system hierarchy of the distributed system, the system hierarchy comprising system domains, each having an active state or an inactive state.

24. The system of claim 23 , wherein the system hierarchy comprises system levels comprising:

a first system level corresponding to host machines of data processing devices, non-transitory memory devices, or network interface controllers, each host machine having a system domain;

a second system level corresponding to power deliverers, communication deliverers, or cooling deliverers of racks housing the host machines, each power deliverer, communication deliverer, or cooling deliverer of the rack having a system domain; and

a third system level corresponding to power deliverers, communication deliverers, or cooling deliverers of cells having associated racks, each power deliverer, communication deliverer, or cooling deliverer of the cell having a system domain; and

a fourth system level corresponding to a distribution center module of the cells, each distribution center module having a system domain.

25. A method comprising:

identifying high-availability jobs and low-availability jobs that demand usage of resources of a distributed system;

determining a first quota of the resources available to low-avai lability jobs as a quantity of the resources available during normal operations;

determining a second quota of the resources available to high-availability jobs based on a resource type, wherein:

for storage capacity resources, the second quota of the resources available to high-availability jobs is the quantity of the resources available during normal operations;

for storage bandwidth resources, the second quota of the resources available to high-availability jobs is the quantity of the resources available during normal operations minus a quantity of the resources lost due to a tolerated event and minus an increased quantity of the remaining resources needed due to the tolerated event; and

for non-storage resource resources, the second quota of the resources available to high-availability jobs is the quantity of the resources available during normal operations minus the quantity of the resources lost due to a tolerated event;

executing the jobs on the resources of the distributed system; and

constraining a total usage of the resources by both the high-availability jobs and the low-availability jobs to the quantity of the resources available during normal operations.

26. The method of claim 25 , further comprising, for non-storage resources, migrating or re-executing jobs assigned to resources lost due to the tolerated event to remaining resources.

27. The method of claim 25 , further comprising, for storage resources:

leaving jobs assigned to the lost resources without reassigning the jobs to remaining resources; and

reconstructing any unavailable data associated with the lost resources.

28. The method of claim 25 , further comprising determining the quantity of the resources lost due to a tolerated event based on an assignment of the jobs to particular resources and a system hierarchy of the distributed system, the system hierarchy comprising system domains, each having an active state or an inactive state.