Systems and methods for fault-tolerant processing with processor regrouping based on connectivity conditions
View Patent ↗Each processor in a distributed multi-processor environment maintains a set of connectivity condition scores. The connectivity condition scores are then passed to a pruning process with tie-breaker logic that selects certain of the processors to include in the new system configuration. The surviving processors are selected based on primary connectivity information and secondary processor connectivity condition scores. The result of the pruning/tie-breaker process is transmitted to all of the surviving processors, which continue to maintain connectivity information that can be used in the next regroup incident.
1. A system for fault-tolerant processing, comprising:
a processor unit;
computer instructions stored on a computer readable medium and operable to:
detect at least one of: failure of other processor units in the system, and connectivity failures that disrupt communications between the processor units;
evaluate connectivity condition scores (CCSs) for the processor units, wherein the processor units are operable to communicate with each other via at least two communication paths, and the CCSs are based on weighted sums of connectivity errors experienced on the communication paths during an observation period;
determine at least two candidate groups with the same number of at least a portion of the processor units to include in the system; and
select between the at least two candidate groups based on the CCSs.
2. The system of claim 1 , wherein the processor units in each candidate group are capable of communicating with the other processor units in the candidate group.
3. The system of claim 1 , wherein the severity of each connectivity error is factored into a corresponding CCS.
4. The system of claim 1 , wherein at least one of the CCSs is based on the history of connectivity errors on the corresponding communication path, and a decay factor is used to define how fast historic CCSs are decayed over an observation period.
5. The system of claim 1 , further comprising:
computer instructions stored on a computer readable medium and operable to:
unpack a bit mask of normalized CCSs from each processor unit.
6. The system of claim 1 , further comprising:
computer instructions stored on a computer readable medium and operable to:
form a bi-directional CCS for each processor unit based on normalized CCSs; and
select between the two candidate groups to include in the system based on the bi-directional CCSs for the processor units in each candidate group.
7. A system for fault-tolerant processing, comprising:
a processor unit configurable to communicate with other components in the system via at least two switching fabrics; and computer instructions stored on a computer readable medium and operable to:
maintain a connectivity condition score (CCS) for each communication path along the at least two fabrics based on connectivity errors experienced on the path, wherein the number of connectivity errors during previous observation time periods are factored into a corresponding CCS during an observation time period, the CCSs are based on weighted sums of connectivity errors experienced on the communication paths during an observation period, and the CCSs are utilized to determine whether the processor unit will continue to be included in the system.
8. The system of claim 7 , wherein the severity of each connectivity error is factored into the corresponding CCS.
9. The system of claim 7 , wherein the processor unit is further configured to communicate the CCSs to at least one of the other components in the system.
10. The system of claim 7 , further comprising:
computer instructions stored on a computer readable medium and operable to: summarize each set of CCSs into a single score.
11. The system of claim 10 , further comprising:
computer instructions stored on a computer readable medium and operable to: normalize each set of CCSs based on the single score.
12. The system of claim 7 , further comprising:
computer instructions stored on a computer readable medium and operable to: transform normalized CCSs into a condensed format.
13. A computer product, comprising:
data structures stored on a computer readable medium including:
a connectivity condition score (CCS) for each communication path associated with a processor unit in a distributed processing system, wherein the CCS indicates the connectivity condition of the communication path during at least one observation period; and
a connectivity matrix indicating whether the processor unit is able to communicate with other components in the system through any of the communication paths; and
a single score representing the sum of the CCSs for the processor unit.
14. The computer product of claim 13 , wherein each CCS is normalized and stored in a bit mask.
15. A method for regrouping processor units in a fault-tolerant system, comprising:
determining the ability of each processor unit to communicate with other processor units in the system;
forming at least two candidate groups with the same number of processor units that are able to communicate with each other;
generating a single score representing a sum of connectivity condition scores (CCSs) for the processor units in the at least two candidate groups, wherein the CCS indicates the connectivity condition of one communication path associated with a corresponding processor unit; and
evaluating the single score for each candidate group of the processor units.
16. The method of claim 15 , wherein the CCS is based on the number of connectivity errors experienced by the corresponding communication path.
17. The method of claim 15 , wherein at least one of the CCSs is based on historical connectivity errors experienced by the corresponding communication path.
18. The method of claim 16 , wherein the severity of each connectivity error is factored into the corresponding CCS.
19. The method of claim 16 , further comprising:
forming a bi-directional CCS for each processor unit; and
selecting between the at least two candidate groups to include in the system based on the sum of the bi-directional CCSs for the processor units in each group.
20. The method of claim 19 , further comprising:
selecting an arbitrary one of the at least two candidate groups when the candidate groups have the same sum of bi-directional CCSs.
21. An apparatus for regrouping processor units in a fault-tolerant system, comprising:
means for forming at least two candidate groups of processor units that are able to communicate with each other; and
means for evaluating connectivity condition scores (CCSs) for each candidate group of the processor units, wherein the number of connectivity errors during previous observation time periods are factored into a corresponding CCS during an observation time period and each CCS indicates the severity of connectivity errors experienced by one communication path associated with a corresponding processor unit; and
means for selecting an arbitrary one of the at least two candidate groups when the candidate groups have the same sum of CCSs.
22. The apparatus of claim 21 , further comprising means for counting the number of connectivity errors experienced by a corresponding communication path during an observation period.
23. The apparatus of claim 21 , further comprising means for factoring into the CCS connectivity errors experienced by a corresponding communication path during at least one previous observation period.
24. The apparatus of claim 21 , further comprising means for selecting a candidate group based on survival priority of the processor units included in each candidate group.
25. The apparatus of claim 24 , further comprising means for selecting a candidate group based on the CCSs, when both candidate groups have the highest number of at least one of the group consisting of: processor units and processor units with the highest survival priority.