Node-local-unscheduler for scheduling remediation
A system for scheduling remediation includes a memory, a processor in communication with the memory, a container scheduled on a first node, a scheduler executing on the processor, and a node-local-unscheduler (“NLU”). The scheduler has a watch module. The NLU executes on the processor to determine a status of the container as failing validation. The NLU has access to scheduling policies corresponding to the container and the first node. Responsive to determining the status of the container as failing validation, the NLU annotates the container and stops execution of the container. The watch module executes on the processor to detect the annotation associated with the container. Responsive to detecting the annotation, the container is rescheduled to a second node.
1. A system comprising:
a memory;
a processor in communication with the memory;
a container scheduled on a first node;
a scheduler executing on the processor to implement scheduling on one or more containers prior to validation when the one or more containers are in a pending state; and
a node-local-unscheduler (NLU) executing on the processor to:
determine a status of the container as failing validation because the container and the first node have conflicting scheduling policies,
responsive to determining the status of the container as failing validation, write an entry relating to the container, and
stop execution of the container,
wherein the processor is configured to:
responsive to detecting the entry, reschedule the container to a second node in a higher priority queue, and
responsive to rescheduling the container, remove the entry.
2. The system of claim 1 , wherein the first node and the second node are in different node clusters.
3. The system of claim 1 , wherein determining the status of the container includes comparing resource requirement metrics of the container with resource availability metrics of the first node, and wherein the container fails validation because the first node has insufficient resources for the container.
4. The system of claim 1 , wherein determining the status of the container includes checking the scheduling policies of the container and operation environment conditions of the container.
5. The system of claim 4 , wherein the operation environment conditions include information about other containers scheduled to the first node.
6. The system of claim 1 , wherein the container is annotated with a flag.
7. The system of claim 1 , wherein the container is in a pod, which is rescheduled to the second node upon failing validation.
8. A method comprising:
scheduling, by a scheduler, a container of one or more containers on a first node;
determining, by a node-local-unscheduler (NLU), a status of the container as failing validation because the container and the first node have conflicting scheduling policies;
responsive to determining the status of the container as failing validation, writing, by the NLU, an entry relating to the container;
stopping, by the NLU, execution of the container;
responsive to detecting the entry, rescheduling the container to a second node in a higher priority queue; and
removing the entry.
9. The method of claim 8 , wherein the first node and the second node are in different node clusters.
10. The method of claim 8 , wherein determining the status of the container includes comparing resource requirement metrics of the container with resource availability metrics of the first node, and wherein the container fails validation because the first node has insufficient resources for the container.
11. The method of claim 8 , wherein determining the status of the container includes checking the scheduling policies of the container and operation environment conditions of the container.
12. The method of claim 11 , wherein the operation environment conditions include information about other containers scheduled to the first node.
13. The method of claim 8 , wherein the container is annotated with a flag.
14. The method of claim 8 , wherein the container is in a pod, which is rescheduled to a second node upon failing validation.
15. A non-transitory machine readable medium storing code, which when executed by one or more processors, causes a node-local-unscheduler (NLU) to:
determine a status of a container scheduled on a first node as failing validation because the container and the first node have conflicting scheduling policies;
responsive to determining the status of the container as failing validation, write an entry relating to the container;
stop execution of the container;
responsive to detecting the entry, reschedule the container to a second node in a higher priority queue; and
responsive to rescheduling the container, remove the entry.
16. The non-transitory machine readable medium of claim 15 , wherein determining the status of the container includes comparing resource requirement metrics of the container with resource availability metrics of the first node, and wherein the container fails validation because the first node has insufficient resources for the container.
17. The non-transitory machine readable medium of claim 15 , wherein determining the status of the container includes checking the scheduling policies of the container and operation environment conditions of the container.
18. The non-transitory machine readable medium of claim 17 , wherein the operation environment conditions include information about other containers scheduled to the first node.
19. The non-transitory machine readable medium of claim 15 , wherein responsive to one or more containers being in a pending state, the code, when executed by the one or more processors, causes a scheduler to implement scheduling of the one or more containers prior to validation.
20. The non-transitory machine readable medium of claim 15 , wherein writing the entry includes annotating a field in container metadata.