OBSERVABILITY AND AUDIT OF AUTOMATIC REMEDIATION OF WORKLOADS IN CONTAINER ORCHESTRATED CLUSTERS

1 . A method of handling a user request to modify state of a container workload in a data center, the method comprising:

receiving the user request at a container orchestrator executing in the data center, the container orchestrator managing the container workload, the container workload executing on a host in the data center;

notifying, by the container orchestrator, a management agent of the user request, the management agent executing in the data center;

receiving, at the container orchestrator from the management agent, an annotated user request and a remediation patch, the annotated user request including metadata describing policies and patches defined in the remediation patch;

applying, by the container orchestrator, the remediation patch to the annotated user request to generate a remediated user request; and

persisting, by the container orchestrator, a state of the container workload in response to the remediated user request.

2 . The method of claim 1 , further comprising:

obtaining, by the management agent from a management service, a policy to be applied to the container workload, the management service executing in a cloud in communication with the data center through a gateway, the gateway executing the management agent;

obtaining, by the management agent, the remediation patch corresponding to the policy; and

adding, by the management agent, the metadata to the user request to generate the annotated user request.

3 . The method of claim 2 , further comprising:

augmenting, by the management agent, the remediation patch with a reversible operation configured to reverse an operation defined in the remediation patch.

4 . The method of claim 1 , wherein the management agent installs a mutating webhook to the container orchestrator and wherein the container orchestrator notifies the management agent of the user request in response to the mutating webhook.

5 . The method of claim 1 , further comprising:

receiving, at the management agent, a notification of the state of the container workload in response to the persisting by the container orchestrator; and

sending, by the management agent to a management service, the state of the container workload, the management service executing in a cloud in communication with the data center through a gateway, the gateway executing the management agent.

6 . The method of claim 5 , further comprising:

decoding, by the management service, the metadata from the remediated user request to compute an original object in the user request;

computing, by the management service, a difference between the original object and a remediated object in the remediated user request; and

persisting the difference in persistent storage of the cloud.

7 . The method of claim 6 , further comprising:

providing, to a user from the management service, the difference between the original object and the remediated object.

8 . A non-transitory computer readable medium comprising instructions to be executed in a computing device to cause the computing device to carry out a method of a method of handling a user request to modify state of a container workload in a data center, the method comprising:

receiving the user request at a container orchestrator executing in the data center, the container orchestrator managing the container workload, the container workload executing on a host in the data center;

notifying, by the container orchestrator, a management agent of the user request, the management agent executing in the data center;

receiving, at the container orchestrator from the management agent, an annotated user request and a remediation patch, the annotated user request including metadata describing policies and patches defined in the remediation patch;

applying, by the container orchestrator, the remediation patch to the annotated user request to generate a remediated user request; and

persisting, by the container orchestrator, a state of the container workload in response to the remediated user request.

9 . The non-transitory computer readable medium of claim 8 , further comprising:

obtaining, by the management agent from a management service, a policy to be applied to the container workload, the management service executing in a cloud in communication with the data center through a gateway, the gateway executing the management agent;

obtaining, by the management agent, the remediation patch corresponding to the policy; and

adding, by the management agent, the metadata to the user request to generate the annotated user request.

10 . The non-transitory computer readable medium of claim 9 , further comprising:

augmenting, by the management agent, the remediation patch with a reversible operation configured to reverse an operation defined in the remediation patch.

11 . The non-transitory computer readable medium of claim 8 , wherein the management agent installs a mutating webhook to the container orchestrator and wherein the container orchestrator notifies the management agent of the user request in response to the mutating webhook.

12 . The non-transitory computer readable medium of claim 8 , further comprising:

receiving, at the management agent, a notification of the state of the container workload in response to the persisting by the container orchestrator; and

sending, by the management agent to a management service, the state of the container workload, the management service executing in a cloud in communication with the data center through a gateway, the gateway executing the management agent.

13 . The non-transitory computer readable medium of claim 12 , further comprising:

decoding, by the management service, the metadata from the remediated user request to compute an original object in the user request;

computing, by the management service, a difference between the original object and a remediated object in the remediated user request; and

persisting the difference in persistent storage of the cloud.

14 . The non-transitory computer readable medium of claim 13 , further comprising:

providing, to a user from the management service, the difference between the original object and the remediated object.

15 . A computing system having a cloud in communication with a data center, the computing system comprising:

a container workload executing in a host of the data center;

a gateway executing in the data center in communication with the cloud, the gateway configured to execute a management agent; and

a container orchestrator executing in the data center configured to manage the container workload, the container orchestrator configured to:

receive a user request;

notify the management agent of the user request;

receive, from the management agent, an annotated user request and a remediation patch, the annotated user request including metadata describing policies and patches defined in the remediation patch;

apply the remediation patch to the annotated user request to generate a remediated user request; and

persisting a state of the container workload in response to the remediated user request.

16 . The computing system of claim 15 , wherein the management agent is configured to:

obtain, from a management service executing in the cloud, a policy to be applied to the container workload;

obtain the remediation patch corresponding to the policy; and

add the metadata to the user request to generate the annotated user request.

17 . The computing system of claim 16 , wherein the management agent is configured to:

augment the remediation patch with a reversible operation configured to reverse an operation defined in the remediation patch.

18 . The computing system of claim 15 , wherein the management agent installs a mutating webhook to the container orchestrator and wherein the container orchestrator notifies the management agent of the user request in response to the mutating webhook.

19 . The computing system of claim 15 , wherein the management agent is configured to:

receive a notification of the state of the container workload in response to the persisting by the container orchestrator; and

send, to a management service, the state of the container workload, the management service executing in a cloud in communication with the data center through a gateway, the gateway executing the management agent.

20 . The computing system of claim 19 , wherein the management service is configured to:

decode the metadata from the remediated user request to compute an original object in the user request;

compute a difference between the original object and a remediated object in the remediated user request; and

persisting the difference in persistent storage of the cloud.