Autoscaling using file access or cache usage for cluster machines

1. A system, comprising:

an interface configured to:

receive an indication to deactivate idle cluster machines of a set of cluster machines; and

a processor configured to:

determine a list of cluster machines storing one or more intermediate data files of a set of intermediate data files;

determine a set of idle cluster machines of the set of cluster machines, wherein the set of idle cluster machines comprises all cluster machines of the set of cluster machines that are neither running one or more tasks of a set of tasks executing or pending on the set of cluster machines nor storing the one or more intermediate data files of the set of intermediate data files, wherein the set of intermediate data files is associated with the set of tasks executing or pending on the set of cluster machines; and

deactivate each cluster machine of the set of idle cluster machines.

2. The system of claim 1 , wherein the set of cluster machines comprise virtual cluster machines.

3. The system of claim 1 , wherein the set of cluster machines comprise physical cluster machines.

4. The system of claim 1 , wherein the processor is further configured to cause a task transfer of a portion of the set of tasks from a first cluster machine to a second cluster machine in order to increase a number of idle cluster machines.

5. The system of claim 1 , wherein the processor is further configured to cause a file transfer of a group of intermediate data files from a first cluster machine to a second cluster machine in order to increase a number of idle cluster machines.

6. The system of claim 1 , wherein the processor is further configured to:

receive a create indication from a cluster machine of the set of cluster machines that an intermediate data file was created; and

add the cluster machine to the list of cluster machines storing intermediate data files in response to the cluster machine not already being on the list of cluster machines storing intermediate data files.

7. The system of claim 6 , wherein determining the set of cluster machines that are not storing the one or more intermediate data files of the set of intermediate data files uses the list of cluster machines storing intermediate data files.

8. The system of claim 6 , wherein the processor is further configured to:

receive a remove indication from the cluster machine that the intermediate data file was removed;

remove the cluster machine from the list of cluster machines storing intermediate data files in response to the cluster machine no longer storing intermediate data files; and

in response to determining that the cluster machine comprises an idle cluster machine in response to removal of the intermediate data file:

deactivate the idle cluster machine.

9. The system of claim 1 , further comprising the set of cluster machines.

10. The system of claim 1 , wherein a first cluster machine of the set of cluster machines comprises a master cluster machine.

11. The system of claim 10 , wherein the allocation system comprises a second cluster machine of the set of cluster machines.

12. The system of claim 1 , wherein the allocation system comprises a system not local to a cluster user system.

13. The system of claim 1 , wherein the allocation system comprises a system local to a cluster user system.

14. The system of claim 1 , wherein the allocation system is part of a cloud-based system.

15. The system of claim 1 , wherein the processor is further configured to determine the set of tasks running on the set of cluster machines, comprising:

providing a query to a master cluster machine for the set of tasks executing or pending on the set of cluster machines; and

receiving the set of tasks executing or pending on the set of cluster machines from the master cluster machine.

16. The system of claim 1 , wherein determining the list of cluster machines storing the one or more intermediate data files of the set of intermediate data files, comprises:

querying data storage associated with each cluster machine of the set of cluster machines; and

creating the list of cluster machines storing intermediate data files associated with tasks of the set of tasks.

17. The system of claim 1 , wherein the list of cluster machines is filtered for intermediate data files that are need for later processing.

18. A method, comprising:

receiving an indication to deactivate idle cluster machines of a set of cluster machines; and

determining, using a processor, a list of cluster machines storing one or more intermediate data files of a set of intermediate data files;

determining a set of idle cluster machines of the set of cluster machines, wherein the set of idle cluster machines comprises all cluster machines of the set of cluster machines that are neither running one or more tasks of a set of tasks executing or pending on the set of cluster machines nor storing the one or more intermediate data files of the set of intermediate data files, wherein the set of intermediate data files are associated with the set of tasks executing or pending on the set of cluster machines; and

deactivating each cluster machine of the set of idle cluster machines.

19. A computer program product, the computer program product being embodied in a non-transitory computer-readable storage medium and comprising computer instructions for:

receiving an indication to deactivate idle cluster machines of a set of cluster machines; and

determining a list of cluster machines storing one or more intermediate data files of a set of intermediate data files;

determining a set of idle cluster machines of the set of cluster machines, wherein the set of idle cluster machines comprises all cluster machines of the set of cluster machines that are neither running one or more tasks of a set of tasks executing or pending on the set of cluster machines nor storing the one or more intermediate data files of the set of intermediate data files, wherein the set of intermediate data files are associated with the set of tasks executing or pending on the set of cluster machines; and

deactivating each cluster machine of the set of idle cluster machines.