Distributed resource scheduling layer utilizable with resource abstraction frameworks
An apparatus in one embodiment comprises a processing platform that includes a plurality of processing devices each comprising a processor coupled to a memory. The processing platform is configured to implement virtual resources for use by client applications. The processing platform is further configured with a multi-layered architecture including at least a client applications layer, a distributed resource scheduling layer, a resource abstraction layer and a resource layer comprising the virtual resources. The distributed resource scheduling layer registers as a framework with the resource abstraction layer, and the client applications register as respective processing tasks with the distributed resource scheduling layer. The distributed resource scheduling layer controls allocation of the virtual resources of the resource layer to the client applications. The virtual resources illustratively comprise respective containers, respective virtual machines, or combinations thereof.
1. An apparatus comprising:
a processing platform comprising a plurality of processing devices each comprising a processor coupled to a memory;
the processing platform being configured to implement virtual resources for use by client applications;
the processing platform being further configured with a multi-layered architecture including at least:
a client applications layer;
a distributed resource scheduling layer;
a resource abstraction layer; and
a resource layer comprising the virtual resources;
the distributed resource scheduling layer being configured to registers as a framework with the resource abstraction layer;
the client applications being configured to register as respective processing tasks with the distributed resource scheduling layer; and
the distributed resource scheduling layer being further configured to control allocation of the virtual resources of the resource layer to the client applications;
the virtual resources comprising first virtual resources of a first type, the first virtual resources of the first type comprising respective container resources implemented using kernel control groups;
the virtual resources further comprising at least second virtual resources of a second type different than the first type, the second virtual resources of the second type comprising respective non-container resources;
the resource abstraction layer being configured to present abstractions of both the first and second virtual resources of the respective first and second types to the distributed resource scheduling layer.
2. The apparatus of claim 1 wherein the virtual resources comprise a plurality of containers allocable to respective ones of the client applications under the control of the distributed resource scheduling layer.
3. The apparatus of claim 2 wherein the distributed resource scheduling layer is configured to pause a given one of the applications utilizing a first container resource and to restart the given application utilizing a second container resource so as to migrate the given application from the first container resource to the second container resource.
4. The apparatus of claim 1 wherein the virtual resources comprise a plurality of virtual machines allocable to respective ones of the client applications under the control of the distributed resource scheduling layer.
5. The apparatus of claim 1 wherein the distributed resource scheduling layer is configured to support a plurality of distinct scheduling algorithms and to dynamically switch between utilization of different ones of the scheduling algorithms in scheduling processing operations for different ones of the client applications.
6. The apparatus of claim 1 wherein the distributed resource scheduling layer is configured to dynamically adjust portions of the virtual resources allocated to different ones of the client applications as a function of relative processing loads of the respective different ones of the client applications.
7. The apparatus of claim 1 wherein the distributed resource scheduling layer is configured to present to the client applications an application programming interface that is substantially the same as that which would otherwise be presented to the client applications by the resource abstraction layer if the distributed resource scheduling layer were removed from the multi-layered architecture.
8. The apparatus of claim 1 wherein the multi-layered architecture further comprises at least one task provider interface arranged between the client applications layer and the distributed resource scheduling layer.
9. The apparatus of claim 1 wherein the multi-layered architecture further comprises at least one resource provider interface arranged between the distributed resource scheduling layer and the resource abstraction layer.
10. The apparatus of claim 1 wherein the resource abstraction layer comprises a plurality of distinct resource abstraction frameworks and further wherein the multi-layered architecture comprises a first resource provider interface arranged between the distributed resource scheduling layer and a first one of the resource abstraction frameworks and a second resource provider interface arranged between the distributed resource scheduling layer and a second one of the resource abstraction frameworks.
11. An information processing system comprising the apparatus of claim 1 .
12. A method comprising:
configuring a processing platform to include a plurality of processing devices each comprising a processor coupled to a memory;
implementing virtual resources within the processing platform for use by client applications;
the processing platform being configured with a multi-layered architecture including at least:
a client applications layer;
a distributed resource scheduling layer;
a resource abstraction layer; and
a resource layer comprising the virtual resources;
the method further comprising:
the distributed resource scheduling layer registering as a framework with the resource abstraction layer;
the client applications registering as respective processing tasks with the distributed resource scheduling layer; and
the distributed resource scheduling layer controlling allocation of the virtual resources of the resource layer to the client applications;
the virtual resources comprising first virtual resources of a first type, the first virtual resources of the first type comprising respective container resources implemented using kernel control groups;
the virtual resources further comprising at least second virtual resources of a second type different than the first type, the second virtual resources of the second type comprising respective non-container resources;
the resource abstraction layer being configured to present abstractions of both the first and second virtual resources of the respective first and second types to the distributed resource scheduling layer.
13. The method of claim 12 further comprising dynamically switching in the distributed resource scheduling layer between utilization of different ones of a plurality of distinct scheduling algorithms supported by that layer in scheduling processing operations for different ones of the client applications.
14. The method of claim 12 further comprising dynamically adjusting in the distributed resource scheduling layer portions of the virtual resources allocated to different ones of the client applications as a function of relative processing loads of the respective different ones of the client applications.
15. The method of claim 12 wherein the virtual resources comprise a plurality of containers allocable to respective ones of the client applications under the control of the distributed resource scheduling layer, the method further comprising:
pausing a given one of the applications utilizing a first container resource; and
restarting the given application utilizing a second container resource;
wherein the pausing and restarting are performed in the distributed resource scheduling layer so as to migrate the given application from the first container resource to the second container resource.
16. The method of claim 12 further comprising:
providing in the multi-layered architecture at least one task provider interface arranged between the client applications layer and the distributed resource scheduling layer; and
providing in the multi-layered architecture at least one resource provider interface arranged between the distributed resource scheduling layer and the resource abstraction layer.
17. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by a processing platform comprising a plurality of processing devices causes the processing platform:
to implement virtual resources for use by client applications;
the processing platform being configured with a multi-layered architecture including at least:
a client applications layer;
a distributed resource scheduling layer;
a resource abstraction layer; and
a resource layer comprising the virtual resources;
the distributed resource scheduling layer being configured to register as a framework with the resource abstraction layer;
the client applications being configured to register as respective processing tasks with the distributed resource scheduling layer; and
the distributed resource scheduling layer being further configured to controls allocation of the virtual resources of the resource layer to the client applications;
the virtual resources comprising first virtual resources of a first type, the first virtual resources of the first type comprising respective container resources implemented using kernel control groups;
the virtual resources further comprising at least second virtual resources of a second type different than the first type, the second virtual resources of the second type comprising respective non-container resources;
the resource abstraction layer being configured to present abstractions of both the first and second virtual resources of the respective first and second types to the distributed resource scheduling layer.
18. The computer program product of claim 17 wherein the program code when executed further causes the processing platform to dynamically switch in the distributed resource scheduling layer between utilization of different ones of a plurality of distinct scheduling algorithms supported by the distributed resource scheduling layer in scheduling processing operations for different ones of the client applications.
19. The computer program product of claim 17 wherein the program code when executed further causes the processing platform to dynamically adjust in the distributed resource scheduling layer portions of the virtual resources allocated to different ones of the client applications as a function of relative processing loads of the respective different ones of the client applications.
20. The computer program product of claim 17 wherein the virtual resources comprise a plurality of containers allocable to respective ones of the client applications under the control of the distributed resource scheduling layer, and wherein the program code when executed further causes the processing platform:
to pause a given one of the applications utilizing a first container resource; and
to restart the given application utilizing a second container resource;
wherein the pausing and restarting are performed in the distributed resource scheduling layer so as to migrate the given application from the first container resource to the second container resource.