Systems and methods for monitoring and maintaining consistency of a configuration
The present application is directed towards systems and methods for monitoring and maintaining consistency of a configuration across a plurality of cores or packet engines in a multi-core system. A configuration manager handles communication of configuration commands to a plurality of cores or packet engines. If a command executes successfully on a first packet engine but fails on a second packet engine, the configuration manager may communicate an undo command to the first packet engine. Successful execution of the undo command may restore the packet engines to a consistent configuration.
1. A method for maintaining consistency of a configuration across a plurality of packet engines, the method comprising:
(a) receiving, by a configuration manager executing on a core of a multi-core system, a set of configuration commands for each of a plurality of packet engines, the set of configuration commands comprising a plurality of configuration commands in a configuration file and each of the plurality of packet engines operating on a corresponding core of a plurality of cores;
(b) propagating, by the configuration manager, one command at a time of the plurality of commands from the set of configuration commands to each of the plurality of packet engines;
(c) determining, by the configuration manager, that a first command of the set of configuration commands executed successfully on each of the plurality of packet engines;
(d) communicating, by the configuration manager responsive to the determination that the first command was executed successfully on each of the plurality of packet engines, a second command of the plurality of commands from the set of commands to each of the plurality of packet engines;
(e) determining, by the configuration manager, that the second command did not execute successfully on at least one packet engine of the plurality of packet engines; and
(f) communicating, by the configuration manager responsive to the determination that the second command did not execute successfully on at least one packet engine, an undo command for the second command to each of the packet engines on which the second command did execute successfully.
2. The method of claim 1 , further comprising:
(g) determining, by the configuration manager, that the undo command did not execute successfully on at least one packet engine of the packet engines that the undo command was communicated to; and
(h) preventing, by the configuration manager, further propagation of commands from the set of configuration commands to each of the plurality of packet engines.
3. The method of claim 2 , wherein (g) further comprises:
determining, by the configuration manager, that the undo command did not execute successfully on at least one packet engine of the plurality of packet engines, responsive to receiving an error code from the at least one packet engine.
4. The method of claim 1 , wherein (b) further comprises:
propagating, by the configuration manager, the first command to a first packet engine;
executing, by the first packet engine, the first command;
determining, by the configuration manager, that the first command was executed successfully on the first packet engine; and propagating, by the configuration manager, the first command to each of the other packet engines of the plurality of packet engines.
5. The method of claim 4 , further comprising:
storing the first command to a configuration store, by the configuration manager, responsive to the determination that the first command was executed successfully on the first packet engine.
6. The method of claim 1 , wherein (e) further comprises:
determining, by the configuration manager, that the second command did not execute successfully on at least one packet engine of the plurality of packet engines, responsive to receiving an error code from the at least one packet engine.
7. The method of claim 1 , further comprising:
maintaining, by each packet engine of the plurality of packet engines, a counter; and
incrementing, by each packet engine, the counter responsive to completing a command successfully.
8. The method of claim 7 , wherein (e) further comprises:
determining, by the configuration manager, that the second command did not execute successfully on at least one packet engine of the plurality of packet engines, responsive to comparing the counters maintained by each packet engine of the plurality of packet engines.
9. The method of claim 1 , wherein the second command is a set command, and further comprising:
retrieving, by the configuration manager prior to communicating the second command, a configuration from a first packet engine of the plurality of packet engines; and
wherein the undo command comprises a set command corresponding to the retrieved configuration.
10. The method of claim 1 , wherein the set of configuration commands comprises one or more configuration commands, and each of the one or more configuration commands includes a counter in a header field set to a value proportional to:
a number of commands in the set of configuration commands, minus one for each prior command in the set of configuration commands.
11. The method of claim 10 , wherein the set of configuration commands comprises an unset group of commands.
12. The method of claim 10 , wherein (a) comprises:
receiving, by the configuration manager, each of the one or more configuration commands of the set of configuration commands in order of execution;
and (b) comprises:
buffering, by the configuration manager, each of the one or more configuration commands until receiving a last configuration command of the set of configuration commands, the last configuration command identified via the counter in the header field of the command; and
communicating, by the configuration manager responsive to receiving the last configuration command, one command at a time from the buffered one or more configuration commands to each of the plurality of packet engines.
13. The method of claim 12 , further comprising:
receiving, by the configuration manager, a second set of configuration commands before completing communicating the one command at a time from the buffered one or more configuration commands; and
buffering the second set of commands.
14. The method of claim 1 , wherein the set of configuration commands is received from a configuration store responsive to an initialization command, and further comprising:
(g) determining, by the configuration manager, that the undo command did not execute successfully on at least one packet engine of the plurality of packet engines;
(h) communicating, by the configuration manager, any remaining commands from the set of configuration commands to each of the plurality of packet engines; and
(i) preventing, by the configuration manager after completing communication of the remaining commands from the set of configuration commands, further propagation of commands to each of the plurality of packet engines.
15. The method of claim 1 , further comprising communicating the first command, by the configuration manager, to a second configuration manager executing on a core of a second multi-core system, responsive to the determination that the first command was executed successfully.
16. A method for monitoring consistency of a configuration across a plurality of packet engines, the method comprising:
(a) obtaining, by a configuration manager executing on a core of a multi-core device, a portion of a live configuration on each of a plurality of packet engines, each of the plurality of packet engines operating on a corresponding core of a plurality of cores of the multi-core device, wherein the obtained portion of the live configuration on each of the plurality of packet engines includes a counter value, incremented for each configuration command successfully executed on the packet engine;
(b) comparing, by the configuration manager, each portion of the live configuration of each packet engine to each portion of the live configuration of the plurality of packet engines;
(c) detecting, by the configuration manager, that at least one portion of the live configuration of a first packet engine of the plurality of packet engines is not consistent with one or more portions of the live configuration of one or more packet engines of the plurality of packet engines;
(d) identifying, by the configuration manager, a difference in the live configuration of the first packet engine detected to be not consistent in comparison to the one or more portions of the live configuration of one or more packet engines; and
(e) communicating, by the configuration manager responsive to identifying the difference, an undo command for a command to each of the packet engines on which the command did execute successfully.
17. The method of claim 16 , further comprising storing an indicator of the identified difference in a storage unit.
18. The method of claim 16 , further comprising reporting the identified difference to an administrator of the multi-core device.
19. The method of claim 16 , further comprising comparing each counter value of each of the plurality of packet engines, and determining, responsive to a difference in the counter.