Reconfiguration framework for byzantine fault tolerant (BFT) state machine replication (SMR) systems
The present disclosure is directed to a novel reconfiguration framework for a BFT SMR system. With this framework, the configuration of both the system itself and the clients of the system can be modified in a live manner (i.e., without taking the system offline) while preserving correct system operation.
1. A method comprising:
receiving, by a replica of a state machine replication (SMR) system, a reconfiguration command directed to the replica;
reaching, by the replica in conjunction with other replicas of the SMR system, a consensus on an execution sequence number for the reconfiguration command; and
executing, by the replica via a reconfiguration engine, the reconfiguration command in accordance with the execution sequence number, the executing of the reconfiguration command comprising applying a configuration change specified in the reconfiguration command to the replica while the SMR system is operational;
wherein the executing includes:
performing a distributed wedge protocol with the other replicas, the distributed wedge protocol causing the replica and the other replicas to agree upon a wedge point at which no further client requests will be processed; and
upon reaching the wedge point and determining that a threshold number of the other replicas have also reached the wedge point, executing the reconfiguration command.
2. The method of claim 1 further comprising:
writing a result or status of the executed reconfiguration command to a storage layer of the replica.
3. The method of claim 1 wherein performing the distributed wedge protocol includes:
designating one replica to submit no-op client requests to the SMR system until all non-faulty replicas have reached the wedge point.
4. The method of claim 1 further comprising:
receiving another reconfiguration command directed to a client of the SMR system; and
writing said another reconfiguration command to a storage layer of the replica.
5. The method of claim 4 wherein in response to the writing, the client:
detects and retrieves said another reconfiguration command from the storage layer;
executes said another reconfiguration command on the client; and
sends a client request to the SMR system for writing a result or status of the executed said another reconfiguration command to the storage layer.
6. The method of claim 1 wherein the reconfiguration engine is distinct from an execution engine of the replica configured to process client requests.
7. A non-transitory computer readable storage medium having stored thereon program code executable by a replica of a state machine replication (SMR) system, the method comprising:
receiving a reconfiguration command directed to the replica;
reaching, in conjunction with other replicas of the SMR system, a consensus on an execution sequence number for the reconfiguration command; and
executing, via a reconfiguration engine, the reconfiguration command in accordance with the execution sequence number, the executing of the reconfiguration command comprising applying a configuration change specified in the reconfiguration command to the replica while the SMR system is operational;
wherein the executing includes:
performing a distributed wedge protocol with the other replicas, the distributed wedge protocol causing the replica and the other replicas to agree upon a wedge point at which no further client requests will be processed; and
upon reaching the wedge point and determining that a threshold number of the other replicas have also reached the wedge point, executing the reconfiguration command.
8. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises:
writing a result or status of the executed reconfiguration command to a storage layer of the replica.
9. The non-transitory computer readable storage medium of claim 7 wherein performing the distributed wedge protocol includes:
designating one replica to submit no-op client requests to the SMR system until all non-faulty replicas have reached the wedge point.
10. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises:
receiving another reconfiguration command directed to a client of the SMR system; and
writing said another reconfiguration command to a storage layer of the replica.
11. The non-transitory computer readable storage medium of claim 10 wherein in response to the writing, the client:
detects and retrieves said another reconfiguration command from the storage layer;
executes said another reconfiguration command on the client; and
sends a client request to the SMR system for writing a result or status of the executed said another reconfiguration command to the storage layer.
12. The non-transitory computer readable storage medium of claim 7 wherein the reconfiguration engine is distinct from an execution engine of the replica configured to process client requests.
13. A computer system acting as a replica in a state machine replication (SMR) system, the computer system comprising:
a processor,
a storage layer; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
receive a reconfiguration command directed to the replica;
reach, in conjunction with other replicas of the SMR system, a consensus on an execution sequence number for the reconfiguration command; and
execute, via a reconfiguration engine, the reconfiguration command in accordance with the execution sequence number, the executing of the reconfiguration command comprising applying a configuration change specified in the reconfiguration command to the replica while the SMR system is operational;
wherein the program code that causes the processor to execute the reconfiguration command includes program code that causes the processor to:
perform a distributed wedge protocol with the other replicas, the distributed wedge protocol causing the replica and the other replicas to agree upon a wedge point at which no further client requests will be processed; and
upon reaching the wedge point and determining that a threshold number of the other replicas have also reached the wedge point, execute the reconfiguration command.
14. The computer system of claim 13 wherein the program code further causes the processor to:
write a result or status of the executed reconfiguration command to the storage layer.
15. The computer system of claim 13 wherein performing the distributed wedge protocol includes:
designating one replica to submit no-op client requests to the SMR system until all non-faulty replicas have reached the wedge point.
16. The computer system of claim 13 wherein the program code further causes the processor to:
receive another reconfiguration command directed to a client of the SMR system; and
write said another reconfiguration command to a storage layer of the replica.
17. The computer system of claim 13 wherein in response to the writing, the client:
detects and retrieves said another reconfiguration command from the storage layer;
executes said another reconfiguration command on the client; and
sends a client request to the SMR system for writing a result or status of the executed said another reconfiguration command to the storage layer.
18. The computer system of claim 13 wherein the reconfiguration engine is distinct from an execution engine of the replica configured to process client requests.