Service level verification in distributed system using data package injection
Methods and systems for managing the operation of a deployment are provided. The deployment may be managed in accordance with a subscription model. The subscription model may use subscriptions to define the extent and limits on use of services provided by the deployment. The subscription services and limits may be enforced by management controllers of network devices of the deployment. The management controllers may operate independently from control planes and data planes of the network devices. If a plane is suspected of being compromised, the management controllers may take action to confirm the suspicions. If confirmed, then remedial activity may be initiated to address the compromised plane.
1. A method of managing operation of a distributed system, the method comprising:
making, by a management controller of a first network device, an identification that a control plane of the first network device is suspected of being compromised;
identifying, by the management controller and based on the identification, a subscription for services provided by the first network device;
obtaining, by the management controller and based on the subscription, a network data package;
injecting, by the management controller, the network data package into a data plane of the first network device to initiate transmission of network data units based on the network data package;
monitoring, by the management controller, processing of the network data units by the data plane to obtain at least one metric indicating a level of network service provided by the first network device;
making a determination, by the management controller, regarding whether the level of the network service is commensurate with the subscription;
in a first instance of the determination where the level of network service is not commensurate with the subscription:
performing, by the management controller, a control plane independent enforcement action to conform operation of the first network device to be commensurate with the subscription to obtain an updated data plane; and
providing, by the updated data plane, network management services to manage network traffic flowing through the distributed system.
2. The method of claim 1 , wherein injecting the network data package comprises:
sending, by the management controller and via a management link, the network data package to a switch application specific integrated circuit of the data plane to initiate processing of the network data units by the data plane.
3. The method of claim 2 , wherein monitoring the processing comprises:
measuring, by the management controller, a duration of time for processing the network data units by the data plane.
4. The method of claim 2 , wherein monitoring the processing comprises:
obtaining, by the management controller, a measurement of a rate at which the network data units are processed by the data plane, the measurement being performed by a second network device.
5. The method of claim 4 , wherein making the determination comprises comparing the measurement to a limit of the subscription corresponding to the rate at which the network data units are processed by the data plane.
6. The method of claim 1 , wherein identifying that the control plane of the first network device is suspected of being compromised comprises:
deploying, by the management controller, a network policy to the control plane; and
monitoring, by the management controller, updates to the data plane to identify whether the data plane is updated by the control plane based on the network policy.
7. The method of claim 6 , wherein the control plane independent enforcement action comprises:
quarantining, by the management controller, the data plane from the control plane; and
updating, by the management controller, operation of the data plane based on the network policy.
8. The method of claim 1 , wherein the management controller comprises a data processing system, the control plane is hosted by computing resources of the first network device, and the data processing system operates independently from the computing resources of the first network device.
9. The method of claim 8 , wherein the management controller is operably connected to the computing resources of the first network device via a first management channel, and the first management channel being usable by the management controller to configure the computing resources of the first network device.
10. The method of claim 9 , wherein the data plane is hosted by a special purpose hardware device operably connected to in-band links through which the network traffic is obtained and forwarded on to other devices, and the management controller is operably connected to at least one other device via an out-of-band link.
11. The method of claim 10 , wherein the management controller is operably connected to the special purpose hardware device via a second management channel, and the second management channel being usable by the management controller to configure the special purpose hardware device.
12. The method of claim 11 , wherein the special purpose hardware device comprises a switch application specific integrated circuit adapted to forward network traffic.
13. The method of claim 1 , wherein the subscription is for a level of service to be provided by the first network device to a subscribing entity.
14. The method of claim 13 , wherein the level of service is a latency level for processing of network data units.
15. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause a network device to perform operations for managing operation of a distributed system, the operations comprising:
making, by a management controller of a first network device, an identification that a control plane of the first network device is suspected of being compromised;
identifying, by the management controller and based on the identification, a subscription for services provided by the first network device;
obtaining, by the management controller and based on the subscription, a network data package;
injecting, by the management controller, the network data package into a data plane of the first network device to initiate transmission of network data units based on the network data package;
monitoring, by the management controller, processing of the network data units by the data plane to obtain at least one metric indicating a level of network service provided by the first network device;
making a determination, by the management controller, regarding whether the level of the network service is commensurate with the subscription;
in a first instance of the determination where the level of network service is not commensurate with the subscription:
performing, by the management controller, a control plane independent enforcement action to conform operation of the first network device to be commensurate with the subscription to obtain an updated data plane; and
providing, by the updated data plane, network management services to manage network traffic flowing through the distributed system.
16. The non-transitory machine-readable medium of claim 15 , wherein injecting the network data package comprises:
sending, by the management controller and via a management link, the network data package to a switch application specific integrated circuit of the data plane to initiate processing of the network data units by the data plane.
17. The non-transitory machine-readable medium of claim 16 , wherein monitoring the processing comprises:
measuring, by the management controller, a duration of time for processing the network data units by the data plane.
18. The non-transitory machine-readable medium of claim 16 , wherein monitoring the processing comprises:
obtaining, by the management controller, a measurement of a rate at which the network data units are processed by the data plane, the measurement being performed by a second network device.
19. A first network device, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the first network device to perform operations for managing operation of a distributed system, the operations comprising:
making, by a management controller of the first network device, an identification that a control plane of the first network device is suspected of being compromised;
identifying, by the management controller and based on the identification, a subscription for services provided by the first network device;
obtaining, by the management controller and based on the subscription, a network data package;
injecting, by the management controller, the network data package into a data plane of the first network device to initiate transmission of network data units based on the network data package;
monitoring, by the management controller, processing of the network data units by the data plane to obtain at least one metric indicating a level of network service provided by the first network device;
making a determination, by the management controller, regarding whether the level of the network service is commensurate with the subscription;
in a first instance of the determination where the level of network service is not commensurate with the subscription:
performing, by the management controller, a control plane independent enforcement action to conform operation of the first network device to be commensurate with the subscription to obtain an updated data plane; and
providing, by the updated data plane, network management services to manage network traffic flowing through the distributed system.
20. The first network device of claim 19 , wherein injecting the network data package comprises:
sending, by the management controller and via a management link, the network data package to a switch application specific integrated circuit of the data plane to initiate processing of the network data units by the data plane.