IP Library Granted Patent US 11,544,228
Granted Patent B2
US 11,544,228 · App. 16/868,874 · Granted Jan 3, 2023

Assignment of quora values to nodes based on importance of the nodes

Inventors: Steven Roscio (Monument, CO); Paul Vencel (Ft. Collins, CO)
Assignee: Hewlett Packard Enterprise Development LP
G06F16/1815G06F11/1451G06F11/1461G06F11/1464G06F11/1469G06F16/178G06F16/182G06F16/1865G06F16/2322G06F16/275
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,544,228
App. No.
16/868,874
Granted
Jan 3, 2023
Kind
B2
Abstract

Embodiments described herein are generally directed to techniques for avoiding or mitigating shared-state damage during a split-brain condition in a distributed network of compute nodes. According to an example, a number, N, of nodes within the distributed computing system is determined. During normal operation of the distributed computing system, a unified state is maintained by synchronizing shared state information. The nodes are ordered by increasing importance to an application from 1 to N. A quora value, q n , is assigned to each of the nodes in accordance with the ordering, where q 1 =1 and each subsequent quora value, q n+1 , is a sum of all prior quora values, q 1 to q n , plus either 1 or a current value of n. These quora values may then be used to determine membership in the dominant or a yielding set to facilitate recovery from the split-brain condition by performing pessimistic or optimistic mitigation actions.

Claims (74)

1. A method comprising:

identifying a number, N, of a plurality of compute nodes within a distributed computing system that runs an application, wherein during normal operation of the distributed computing system a unified state is maintained by synchronizing shared state information among the plurality of compute nodes;

ordering the plurality of compute nodes in order of increasing importance to the application from 1 to N, the importance to the application being based on one or more of compute capacity, compute performance, system or environment robustness of a compute node;

assigning a quora value, q n , to each of the plurality of compute nodes in accordance with the ordering, wherein q 1 =1 and each subsequent quora value, q n+ 1, is a sum of all prior quora values, q 1 to q n , plus either 1 or a current value of n;

determining, by a node of the plurality of compute nodes, existence of a split-brain condition; and

performing a mitigation approach,

wherein said determining, by a node of the plurality of compute nodes, existence of a split-brain condition, comprises:

determining, by the node, which of the plurality of compute nodes are reachable by the node by probing all other nodes of the plurality of compute nodes;

dynamically determining, by the node, a current pretense value of the node, wherein the current pretense value is a sum of all quora values of a subset of the plurality of compute nodes that are currently reachable by the node, plus the quora value of the compute node; and

when the current pretense value is less than a quorum for the distributed computing system, recognizing the existence of the split-brain condition.

2. The method of claim 1 , further comprising determining by a particular node of the plurality of compute nodes whether the particular node is part of a dominant set of the plurality of compute nodes or part of a yielding set of the plurality of compute nodes.

3. The method of claim 1 , further comprising responsive to a result of said determining, by a node of the plurality of compute nodes, existence of a split-brain condition being affirmative, discontinuing, by the node, further processing of transactions relating to the application.

4. The method of claim 1 , further comprising responsive to a result of said determining, by a node of the plurality of compute nodes, existence of a split-brain condition being affirmative, performing an optimistic mitigation approach by:

creating, by the node, a checkpoint; and

continuing, by the node, processing of transactions relating to the application, including journaling of the transactions by persisting a current pretense value of the node with each of the transactions, and optionally including an identity of a user causing the transaction, and optionally notifying the user that the transaction is pending a future mitigation.

5. The method of claim 4 , further comprising responsive to resolution of the split-brain condition, resolving, by the node, any conflicting transactions of the journaled transactions by:

reverting to the checkpoint;

retrieving journals from other nodes of the plurality of compute nodes;

sort-merging transactions performed subsequent to the checkpoint;

replaying the sort-merged transactions;

when said replaying identifies conflicting transactions for a particular record, retaining a first transaction of the conflicting transactions having a highest pretense value and discarding a second transaction of the conflicting transactions having a lower pretense value and an earlier timestamp; and

when said replaying does not identify any conflicting transactions for a particular transaction associated with the particular record, then committing the particular transaction.

6. The method of claim 1 , wherein ordering the plurality of compute nodes in order of increasing importance to the application from 1 to N comprises generating a sorted list in which a first compute node in the list is of least importance to the application and a last compute node in the list is of greatest importance to the application.

7. The method of claim 1 , wherein identifying a number, N, of a plurality of compute nodes within a distributed computing system that runs an application comprises receiving first configuration information associated with the distributed system.

8. The method of claim 1 , wherein identifying a number, N, of a plurality of compute nodes within a distributed computing system that runs an application comprises receiving first configuration information associated with the distributed system.

9. A non-transitory machine readable medium storing instructions executable by a processing resource of a computer system, the non- transitory machine readable medium comprising instructions to:

identify a number, N, of a plurality of compute nodes within a distributed computing system that runs an application, wherein during normal operation of the distributed computing system a unified state is maintained by synchronizing shared state information among the plurality of compute nodes;

order the plurality of compute nodes in order of increasing importance to the application from 1 to N, the importance to the application being based on one or more of compute capacity, compute performance, system or environment robustness of a compute node;

assign a quora value, q n , to each of the plurality of compute nodes in accordance with the ordering, wherein q 1= 1 and each subsequent quora value, q n+ 1, is a sum of all prior quora values, q 1 to q n , plus either 1 or a current value of n;

determine, by a node of the plurality of compute nodes, existence of a split-brain condition; and

perform a mitigation approach,

wherein the determination regarding the existence of the split-brain condition comprises instructions to:

determine, by the node, which of the plurality of compute nodes are reachable by the node by probing all other nodes of the plurality of compute nodes;

dynamically determine, by the node, a current pretense value of the node, wherein the current pretense value is a sum of all quora values of a subset of the plurality of compute nodes that are currently reachable by the node, plus the quora value of the compute node; and

when the current pretense value is less than a quorum for the distributed computing system, recognize the existence of the split-brain condition.

10. The non-transitory machine readable medium of claim 9 , wherein the instructions further comprise instructions to determine by a particular node of the plurality of compute nodes whether the particular node is part of a dominant set of the plurality of compute nodes or part of a yielding set of the plurality of compute nodes.

11. The non-transitory machine readable medium of claim 9 , wherein the instructions further comprise instructions to responsive to determining existence of the split-brain condition, discontinue, by the node, further processing of transactions relating to the application.

12. The non-transitory machine readable medium of claim 9 , wherein the instructions further comprise instructions to responsive to determining existence of the split-brain condition, perform an optimistic mitigation approach by:

creating, by the node, a checkpoint; and

continuing, by the node, processing of transactions relating to the application, including journaling, the transactions by persisting a current pretense value of the node with each of the transactions.

13. The non-transitory machine readable medium of claim 12 , wherein the instructions further comprise instructions to responsive to resolution of the split-brain condition, resolve, by the node, any conflicting transactions of the journaled transactions by:

reverting to the checkpoint;

retrieving journals from other nodes of the plurality of compute nodes;

sort-merging transactions performed subsequent to the checkpoint;

replaying the sort-merged transactions;

when said replaying identifies conflicting transactions for a particular record, retaining a first transaction of the conflicting transactions having a highest pretense value and discarding a second transaction of the conflicting transactions having a lower pretense value and an earlier timestamp; and

when said replaying does not identify any conflicting transactions for a particular transaction associated with the particular record, then committing the particular transaction.

14. The method of claim 9 , wherein ordering the plurality of compute nodes in order of increasing importance to the application from 1 to N comprises generating a sorted list in which a first compute node in the list is of least importance to the application and a last compute node in the list is of greatest importance to the application.

15. A system comprising:

a processing resource; and

a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that, when executed by the processing resource, cause the processing resource to:

identify a number, N, of a plurality of compute nodes within a distributed computing system that runs an application, wherein during normal operation of the distributed computing system a unified state is maintained by synchronizing shared state information among the plurality of compute nodes;

order the plurality of compute nodes in order of increasing importance to the application from 1 to N, the importance to the application being based on one or more of compute capacity, compute performance, system or environment robustness of a compute node;

assign a quora value, q n , to each of the plurality of compute nodes in accordance with the ordering, wherein q 1= 1 and each subsequent quora value, q n +1, is a sum of all prior quora values, q 1 to q n , plus either 1 or a current value of n;

determine existence of a split-brain condition; and

perform a mitigation approach,

wherein the determination regarding the existence of the split-brain condition comprises instructions that cause the processing resource to:

determine which of the plurality of compute nodes are reachable by the node by probing all other nodes of the plurality of compute nodes;

dynamically determine a current pretense value of the node,

wherein the current pretense value is a sum of all quora values of a subset of the plurality of compute nodes that are currently reachable by the node, plus the quora value of the compute node; and

when the current pretense value is less than a quorum for the distributed computing system, recognize the existence of the split-brain condition.

16. The system of claim 15 , wherein the instructions further cause the processing resource to determine by a particular node of the plurality of compute nodes whether the particular node is part of a dominant set of the plurality of compute nodes or part of a yielding set of the plurality of compute nodes.

17. The system of claim 15 , wherein the instructions further cause the processing resource to instructions to responsive to determining existence of the split-brain condition, perform an optimistic mitigation approach by:

creating, by the node, a checkpoint; and

continuing, by the node, processing of transactions relating to the application, including journaling, the transactions by persisting a current pretense value of the node with each of the transactions.

18. The system of claim 17 , wherein the instructions further cause the processing resource to responsive to resolution of the split-brain condition, resolve, by the node, any conflicting transactions of the journaled transactions by:

reverting to the checkpoint;

retrieving journals from other nodes of the plurality of compute nodes;

sort-merging transactions performed subsequent to the checkpoint;

replaying the sort-merged transactions;

when said replaying identifies conflicting transactions for a particular record, retaining a first transaction of the conflicting transactions having a highest pretense value and discarding a second transaction of the conflicting transactions having a lower pretense value and an earlier timestamp; and

when said replaying does not identify any conflicting transactions for a particular transaction associated with the particular record, then committing the particular transaction.

19. The method of claim 15 , wherein ordering the plurality of compute nodes in order of increasing importance to the application from 1 to N comprises generating a sorted list in which a first compute node in the list is of least importance to the application and a last compute node in the list is of greatest importance to the application.

20. The method of claim 15 , wherein the instructions further cause the processing resource to, responsive to determining existence of the split-brain condition, discontinue, by the node, further processing of transactions relating to the application.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 11, 2020
From: ROSCIO, STEVE; VENCEL, PAUL
To: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP
Reel/Frame 052625/0125 →
Continuity (1)
Related Publication 20210349860A1 · Nov 11, 2021