Network operating system for maintaining redundant master control blade management information

1. A method comprising:

executing a network operating system (NOS) on each processor element of a switch;

via the NOS, creating a distributed messaging layer channel among a dynamic set of control blades of a plurality of control blades of the switch that are present and intercommunicating in the switch;

maintaining master control blade management information, including an object manager global database containing information regarding and configuration of active virtual private networks (VPNs) within the switch, active virtual routers (VRs) within the switch, object groups within the switch and objects within the switch, on a master control blade of the dynamic set of control blades;

maintaining redundant master control blade management information, including a replica of the object manager global database, on one or more standby control blades of the dynamic set of control blades by

the distributed messaging layer channel notifying the master control blade and the one or more standby control blades of detection of dynamic events within the switch, including (i) a peer up event, indicating a new control blade of the dynamic set of control blades has become reachable, (ii) a peer down event, indicating one of the dynamic set of control blades has become unreachable, (iii) a master up event, indicating a new master control blade has been elected, and (iv) a master down event, indicating the master control blade has become unreachable,

performing bulk updates and flash updates to the redundant master control blade management information, the bulk updates being triggered by the dynamic events, the flash updates being triggered by individual changes to the object manager global database, including creation or deletion of a virtual router;

each of the one or more standby control blades periodically performing a consistency check against the replica of the object manager global database; and

if the consistency check indicates one or more of a plurality of consistency rules, including one or more rules relating to internal consistency of the redundant master control blade management information, has failed, then the standby control blade requesting a bulk update from the master control blade.

2. The method of claim 1 , wherein the plurality of consistency rules includes rules verifying one or more of the following conditions:

for every virtual private network (VPN) there is a unique ID within the switch;

for every virtual router (VR) there is a unique combination of a VPN ID and a VR ID;

every VR has a unique VR ID within its associated VPN;

for every object group there is a VR to which the object group belongs;

for every object there is an object group to which the object belongs;

every object has a unique object ID in the switch;

for each VPN, a value of a counter tracking a total number of VRs associated with the VPN is equal to an actual total number of VRs associated with the VPN;

for each object group, a value of a counter tracking a number of objects in the object group is equal to an actual total number of objects in the object group;

a total number of objects across all object groups is equal to a total number of objects across address spaces and is equal to a total number of objects in the switch;

a total number of objects in a sorted global list is equal to a total number of objects in a global hash and is equal to the total number of objects in the switch; and

for every object class in the switch, there is a corresponding entry in a class table.

3. A switch comprising:

a master control blade having stored thereon an object manager global database containing information regarding virtual private network (VPN) and virtual router (VR) configuration as well as global objects and global object groups, wherein objects represent a basic unit of management within the switch for purposes of fault tolerance;

a plurality of standby control blades each having stored thereon a replica of the object manager global database; and

a separate instance of a network operating system (NOS) running on the master control blade and on each of the plurality of standby control blades, the NOS including

an object manager component concerned with maintaining the object manager global database;

a control blade redundancy module, which creates and maintains the plurality of replicas by sending updates to each of the plurality of replicas responsive to one or more predefined dynamic events within the switch and changes to the object manager global database; and

wherein the NOS is operative to

notify the master control blade and the plurality of standby control

blades of detection of the one or more predefined dynamic events including (i) a peer up even, indicating a new control blade has become reachable, (ii) a peer down event, indicating one of the plurality of standby control blades has become unreachable, (iii) a master up event, indicating a new master control blade has been elected and (iv) a master down event indicating the master control blade has become unreachable;

cause bulk updates and flash updates to be performed for the plurality of replicas, the bulk updates being triggered by the one or more predefined dynamic events, the flash updates being triggered by individual changes to the object manager global database, including creation or deletion of a virtual router within the switch;

cause a set of consistency checks to be performed periodically on the plurality of replicas; and

if the set of consistency checks indicates internal inconsistency of a replica of the plurality of replicas, then initiating a bulk update of the replica from the object manager global database.

4. The switch of claim 3 , wherein the NOS further comprises a distributed messaging layer which notifies the control blade redundancy module of events within the switch including the one or more predefined dynamic events.

5. The switch of claim 3 , wherein the plurality of replicas are synchronized at least in part by receiving the bulk updates responsive to the peer up event or the peer down event.

6. The switch of claim 3 , wherein the plurality of replicas are synchronized at least in part by receiving the flash updates used to propagate individual changes.

7. The switch of claim 3 , wherein the one or more predefined dynamic events include virtual router change events indicative of creation of a new virtual router operable within the switch or deletion of an existing virtual router operable within the switch and wherein one or more flash updates are triggered responsive to the virtual router change events.

8. The switch of claim 3 , wherein each of the plurality of standby control blades periodically performs the set of consistency checks to verify one or more of the following conditions:

for every virtual private network (VPN) there is a unique ID within the switch;

for every virtual router (VR) there is a unique combination of a VPN ID and a VR ID;

every VR has a unique VR ID within its associated VPN;

for every object group there is a VR to which the object group belongs;

for every object there is an object group to which the object belongs;

every object has a unique object ID in the switch;

for each VPN, a value of a counter tracking a total number of VRs associated with the VPN is equal to an actual total number of VRs associated with the VPN;

for each object group, a value of a counter tracking a number of objects in the object group is equal to an actual total number of objects in the object group;

a total number of objects across all object groups is equal to a total number of objects across address spaces and is equal to a total number of objects in the switch;

a total number of objects in a sorted global list is equal to a total number of objects in a global hash and is equal to the total number of objects in the switch; and

for every object class in the switch, there is a corresponding entry in a class table.

9. The switch of claim 8 , wherein if results of the set of consistency checks for a standby control blade of the plurality of standby control blades indicates a need for resynchronization, then the replica associated with the standby control blade is updated from the master control blade.

10. A program storage device readable by a switch, tangibly embodying a program of instructions executable by a plurality of processor elements of the switch to perform method steps for maintaining redundant master control blade management information, said method steps comprising:

executing a network operating system (NOS) on each processor element of the plurality of processor elements;

via the NOS, creating a distributed messaging layer channel among a dynamic set of control blades of a plurality of control blades of the switch that are present and intercommunicating in the switch;

maintaining master control blade management information, including an object manager global database containing information regarding and configuration of active virtual private networks (VPNs) within the switch, active virtual routers (VRs) within the switch, object groups within the switch and objects within the switch, on a master control blade of the dynamic set of control blades;

maintaining redundant master control blade management information, including a replica of the object manager global database, on one or more standby control blades of the dynamic set of control blades by

the distributed messaging layer channel notifying the master control blade and the one or more standby control blades of detection of dynamic events within the switch, including (i) a peer up event, indicating a new control blade of the dynamic set of control blades has become reachable, (ii) a peer down event, indicating one of the dynamic set of control blades has become unreachable, (iii) a master up event, indicating a new master control blade has been elected, and (iv) a master down event, indicating the master control blade has become unreachable,

performing bulk updates and flash updates to the redundant master control blade management information, the bulk updates being triggered by the dynamic events, the flash updates being triggered by individual changes to the object manager global database, including creation or deletion of a virtual router;

each of the one or more standby control blades periodically performing a consistency check against the replica of the object manager global database; and

if the consistency check indicates one or more of a plurality of consistency rules, including one or more rules relating to internal consistency of the redundant master control blade management information, has failed, then the standby control blade requesting a bulk update from the master control blade.

11. The program storage device of claim 10 , wherein the plurality of consistency rules includes rules verifying the following conditions:

for every virtual private network (VPN) there is a unique ID within the switch;

for every virtual router (VR) there is a unique combination of a VPN ID and a VR ID;

every VR has a unique VR ID within its associated VPN;

for every object group there is a VR to which the object group belongs;

for every object there is an object group to which the object belongs; and

every object has a unique object ID in the switch.

12. The program storage device of claim 10 , wherein the plurality of consistency rules includes rules verifying the following conditions:

for each object group, a value of a counter tracking a number of objects in the object group is equal to an actual total number of objects in the object group;

a total number of objects across all object groups is equal to a total number of objects across address spaces and is equal to a total number of objects in the switch;

a total number of objects in a sorted global list is equal to a total number of objects in a global hash and is equal to the total number of objects in the switch; and

for every object class in the switch, there is a corresponding entry in a class table.