System and method of fault tolerant reconciliation for control card redundancy

1. A method of redundant control plane synchronization in network node equipment having an active control plane and an inactive control plane, wherein the active control plane comprises an active control plane synchronization error logic module coupled to an error log and an active control plane database, and wherein the inactive control plane comprises an inactive control plane synchronization error logic module coupled to a tolerable attribute error list and an inactive control plane database, the method comprising:

synchronizing a plurality of attributes between the inactive control plane database and the active control plane database by updating the inactive control plane database to mirror the active control plane database;

detecting at least one error in the synchronization of said inactive control plane database and said active control plane database and sending the at least one error to the error log;

using the inactive control plane synchronization error logic module to identify said at least one error and identify which of said attributes of said inactive;

control plane database and said active control plane database caused said at least one error;

using the inactive control plane synchronization error logic module to compare attributes that caused said at least one error to entries in the tolerable attribute error list;

when any of said attributes that caused said at least one error are not in said tolerable attribute error list, determining the inactive control plane database to be intolerable for a redundant switchover, and executing a hard restart with said inactive control plane database;

when all of said attributes that caused said at least one error are in said tolerable attribute error list, determining the inactive control plane database to be tolerable for said redundant switchover unless a maximum number of tolerable attribute errors has been reached; and

performing a second synchronization, the second synchronization comprising:

using the active control plane synchronization error logic module to make a back-up error log of the error log,

synchronizing the active and inactive control plane databases to create a new error log, and comparing the back-up and new error logs, so that when the comparison is inconsistent, a hard reset occurs on the inactive control plane, and when the comparison is consistent, the inactive control plane continues to operate and an operator is notified that the inactive control plane database is in a compromised state;

determining an error code associated with an occurrence of said at least one error; and

determining a state of the inactive control plane database in the context of the at least one error to be compromised less than a maximum tolerable magnitude of compromization.

2. The method of claim 1 , further comprising:

evaluating a logical qualifier with use of said at least one error and said state of the inactive control plane database as inputs.

3. The method of claim 1 , further comprising:

populating said tolerable attribute error list with entries corresponding to:

attributes of said plurality of attributes for which errors in said inactive control plane database would cause a limited impact on services provided by the network node equipment, and

services provided by a network in which said network node equipment is situated, should a redundant switchover to the inactive control plane occur.

4. The method of claim 3 , further comprising:

determining that a number of total attributes of said plurality of attributes which have been found in said tolerable attribute error list is less than a predetermined maximum number of attribute errors.

5. A system for redundant control plane synchronization in network node equipment, the system comprising:

a memory;

an inactive control plane comprising: an inactive control plane database, an inactive control plane synchronization error logic module coupled to a tolerable error attribute list, and an inactive control plane synchronization module;

an active control plane coupled to said inactive control plane by a synchronization connection, said active control plane comprising:

an active control plane database, an active control plane synchronization error logic module coupled to an error log, an active control plane synchronization module that synchronizes a plurality of attributes between said inactive control plane database and said active control plane database in cooperation with said inactive control plane synchronization module by updating the inactive control plane database to mirror the active control plane database, wherein the inactive control plane synchronization error logic module identifies at least one error and at least one of the plurality of attributes which has caused the at least one error,

compares attributes that caused said at least one error to entries in the tolerable attribute error list,

when any of said attributes that caused said at least one error are not in said tolerable attribute error list, determines the inactive control plane database to be intolerable for a redundant switchover and executes a hard restart with said inactive control plane database,

when all of said attributes that caused said at least one error are on said tolerable attribute error list, determines the inactive control plane database to be tolerable for said redundant switchover unless a maximum number of tolerable attribute errors has been reached, and

wherein the active control plane synchronization error logic module makes a back-up error log of the error log and,

after the active and inactive control plane database are synchronized to create a new error log, compares the back-up and new error logs, so that when the comparison is inconsistent, a hard reset occurs on the inactive control plane,

when the comparison is consistent, the inactive control plane continues to operate and an operator is notified that the inactive control plane database is in a compromised state, wherein said inactive control plane synchronization error logic module identifies the at least one error by determining an error code associated with an occurrence of said at least one error, wherein said inactive control plane synchronization error logic module determines the inactive control plane database to be tolerable by determining a state of the inactive control plane database in the context of the at least one error to be compromised less than a maximum tolerable magnitude of compromization.

6. The system of claim 5 , wherein said inactive control plane synchronization error logic module determines the state of the inactive control plane database in the context of the at least one error to be compromised less than the maximum tolerable magnitude of compromization by evaluating a logical qualifier with use of said at least one error and said state of the inactive control plane database as inputs.

7. The system of claim 5 , wherein said tolerable attribute error list is populated with entries corresponding to attributes of said plurality of attributes for which errors in said inactive control plane database would cause a limited impact on services provided by the network node equipment, and services provided by a network in which said network node equipment is situated, should a redundant switchover to the inactive control plane occur.

8. The system of claim 7 , wherein said inactive control plane synchronization error logic module determines the inactive control plane database to be tolerable by also determining that a number of total attributes of said plurality of attributes which have been found in said tolerable attribute error list is less than a predetermined maximum number of tolerable attribute errors.