Autoscaling using file access or cache usage for cluster machines

1. An allocation system, comprising:

an interface configured to receive an indication to deactivate idle cluster machines of a set of cluster machines; and

a processor configured to:

determine a set of tasks executing or pending on the set of cluster machines;

determine a set of idle cluster machines of the set of cluster machines, wherein the set of idle cluster machines comprises all cluster machines of the set of cluster machines that are neither running one or more tasks of the set of tasks nor storing one or more intermediate data files of a set of intermediate data files, wherein the set of intermediate data files is associated with set of tasks executing or pending on the set of cluster machines; and

deactivate each cluster machine of the set of idle cluster machines.

2. The allocation system of claim 1 , wherein the cluster machines comprise virtual cluster machines.

3. The allocation system of claim 1 , wherein the cluster machines comprise physical cluster machines.

4. The allocation system of claim 1 , wherein the processor is further configured to cause a task transfer of a portion of the set of tasks from a first cluster machine to a second cluster machine in order to increase a number of idle cluster machines.

5. The allocation system of claim 1 , wherein the processor is further configured to cause a file transfer of a group of intermediate data files from a first cluster machine to a second cluster machine in order to increase a number of idle cluster machines.

6. The allocation system of claim 1 , wherein the processor is further configured to:

receive an indication from a cluster machine that an intermediate data file was created; and

add the cluster machine to a list of cluster machines storing intermediate data files in the event that the cluster machine is not already on the list.

7. The allocation system of claim 6 , wherein determining a set of cluster machines that are not storing the one or more intermediate data files of the set of intermediate data files uses the list of cluster machines storing intermediate data files.

8. The allocation system of claim 6 , wherein the processor is further configured to:

receive an indication from the cluster machine that the intermediate data file was removed;

remove the cluster machine from the list of cluster machines storing intermediate data files in the event that the cluster machine no longer stores intermediate data files; and

in response to a determination that a cluster machine comprises an idle cluster machine in response to the removal of the intermediate data file:

deactivate the idle cluster machine.

9. The allocation system of claim 1 , further comprising the set of cluster machines.

10. The allocation system of claim 1 , wherein a first cluster machine of the set of cluster machines comprises a master cluster machine.

11. The allocation system of claim 10 , wherein the allocation system comprises a second cluster machine of the set of cluster machines.

12. The allocation system of claim 1 , wherein the allocation system comprises a system not local to a cluster user system.

13. The allocation system of claim 1 , wherein the allocation system comprises a system local to a cluster user system.

14. The allocation system of claim 1 , wherein the allocation system is part of a cloud-based system.

15. The allocation system of claim 1 , wherein the processor is further configured to determine the set of tasks running on the set of cluster machines, comprising:

providing a query to a master cluster machine for the set of tasks executing or pending on the set of cluster machines; and

receiving the set of tasks executing or pending on the set of cluster machines from the master cluster machine.

16. The allocation system of claim 1 , wherein the processor is further configured to determine a list of cluster machines storing the one or more intermediate data files of the set of intermediate data files, comprising:

querying data storage associated with each cluster machine of the set of cluster machines; and

creating a list of cluster machines storing intermediate data files associated with tasks of the set of tasks.

17. The allocation system of claim 1 , wherein the list of cluster machines is filtered for intermediate data files that are need for later processing.

18. A method for allocation, comprising:

receiving an indication to deactivate idle cluster machines of a set of cluster machines; and

determining, using a processor, a set of tasks executing or pending on the set of cluster machines;

determining a set of idle cluster machines of the set of cluster machines, wherein the set of idle cluster machines comprises all cluster machines of the set of cluster machines that are neither running one or more tasks of the set of tasks nor storing one or more intermediate data files of a set of intermediate data files, wherein the set of intermediate data files are associated with the set of tasks executing or pending on the set of cluster machines; and

deactivating each cluster machine of the set of idle cluster machines.

19. A computer program product for allocation, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving an indication to deactivate idle cluster machines of a set of cluster machines; and

determining a set of tasks executing or pending on the set of cluster machines;

determining a set of idle cluster machines of the set of cluster machines, wherein the set of idle cluster machines comprises all cluster machines of the set of cluster machines that are neither running one or more tasks of the set of tasks nor storing one or more intermediate data files of a set of intermediate data files, wherein the set of intermediate data files are associated with the set of tasks executing or pending on the set of cluster machines; and

deactivating each cluster machine of the set of idle cluster machines.