Method and apparatus for switching service nodes in a distributed storage system

1. A method comprising:

initializing, by a storage system, at least one checking thread;

issuing, by the storage system via the checking thread, service requests to a plurality of service nodes, the service nodes including a current service node and one or more backup service nodes;

storing, by the storage system, responses to the service requests in a service request queue;

monitoring, by the storage system, response states of a plurality of service nodes by analyzing the service request queue to identify response abnormalities;

halting, by the storage system, communication between the current service node and the backup service nodes in response to detecting an abnormal response state of the current service node; and

triggering, by the storage system, a switching process for the current service node, the switching process comprising replacing the current service node with a service node selected from the backup service nodes.

2. The method of claim 1 , the initializing, by the storage system, at least one checking thread comprising initializing a plurality of checking threads each checking thread associated with checkpoints associated with a response state.

3. The method of claim 1 , the initializing at least one checking thread comprising initializing, by the storage system, a lightweight thread, the lightweight thread prevented from performing computationally-intensive operations.

4. The method of claim 1 , the halting communication between the current service node and the backup service nodes comprising one or more of:

halting, by the storage system, heartbeat acknowledgements between the current service node and the backup service nodes;

releasing, by the storage system, a lock held by the current service node; and

assigning, by the storage system, an abnormal flag to the current service node.

5. The method of claim 1 , the replacing the current service node with a service node selected from the backup service nodes comprising replacing, by the storage system, the current switching service node with a service node selected from the backup service nodes using a Paxos protocol.

6. The method of claim 1 , the monitoring response states further comprising:

storing, by the storage system, a monitoring result corresponding the response states in a monitoring result registration list; and

polling, by a background thread executed by the storage system, the monitoring result registration list.

7. The method of claim 6 , the polling the monitoring result registration list comprising polling, by the storage system, the monitoring result registration list at a predefined interval, the predefined interval shorter than a preset response time threshold for monitoring the response states.

8. The method of claim 1 , the monitoring response states further comprising:

acquiring, through a first checking thread executed by the storage system, a time point (T 1 ) at which a service node last fetches the service request from the service request queue;

using, by the first checking thread executed by the storage system, a time interval (D 1 ) between a current time point (N 1 ) and the time point (T 1 ) as a response time; and

determining, by the first checking thread executed by the storage system, that the response state of the service node is abnormal if the response time is greater than a first preset response time threshold.

9. The method of claim 8 , the monitoring response states further comprising:

determining, through a second checking thread executed by the storage system, that a storage unit of the service node includes a storage response timeout flag;

using, by the storage system, a time interval (D 2 ) between a flag-adding time point (T 2 ) corresponding to the storage response timeout flag and a current time point (N 2 ) as the response time; and

determining, by the storage system, that the response state of the service node is abnormal if the response time is greater than a second preset response time threshold.

10. The method of claim 9 , wherein, prior to the monitoring response states, the method further comprises:

acquiring, through a log recording thread executed by the storage system, start and end times of log writing of the storage unit of the service node;

using, by the storage system, a time interval between the start and end times as a storage unit response time;

adding, by the storage system, the storage response timeout flag to the storage unit; and

recording, by the storage system, a time point (T 2 ) when the flag is added if the storage unit response time is greater than a third preset response time threshold.

11. An apparatus comprising:

a processor; and

a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic comprising:

logic, executed by the processor, for initializing at least one checking thread;

logic, executed by the processor, for issuing, via the checking thread, service requests to a plurality of service nodes, the service nodes including a current service node and one or more backup service nodes;

logic, executed by the processor, for storing responses to the service requests in a service request queue;

logic, executed by the processor, for monitoring response states of a plurality of service nodes by analyzing the service request queue to identify response abnormalities;

logic, executed by the processor, for halting communication between the current service node and the backup service nodes in response to detecting an abnormal response state of the current service node; and

logic, executed by the processor, for triggering a switching process for the current service node, the switching process comprising replacing the current service node with a service node selected from the backup service nodes.

12. The apparatus of claim 11 , the logic for initializing the at least one checking thread comprising logic, executed by the processor, for initializing a plurality of checking thread, each checking thread associated with checkpoints associated with a response state.

13. The apparatus of claim 11 , the logic for initializing at least one checking thread comprising logic, executed by the processor, for initializing a lightweight thread, the lightweight thread prevented from performing computationally-intensive operations.

14. The apparatus of claim 11 , the logic for halting communication between the current service node and the backup service nodes comprising one or more of:

logic, executed by the processor, for halting heartbeat acknowledgements between the current service node and the backup service nodes;

logic, executed by the processor, for releasing a lock held by the current service node; and

logic, executed by the processor, for assigning an abnormal flag to the current service node.

15. The apparatus of claim 11 , the logic for replacing the current service node with a service node selected from the backup service nodes comprising logic, executed by the processor, for replacing the current service node with a service node selected from the backup service nodes using a Paxos protocol.

16. The apparatus of claim 11 , the logic for monitoring response states further comprising:

logic, executed by the processor, for storing a monitoring result corresponding the response states in a monitoring result registration list; and

logic, executed by the processor, for polling, via a background thread, the monitoring result registration list.

17. The apparatus of claim 16 , the logic for polling the monitoring result registration list comprising logic, executed by the processor, for polling the monitoring result registration list at a predefined interval, the predefined interval shorter than a preset response time threshold for monitoring the response states.

18. The apparatus of claim 11 , the logic for monitoring response states further comprising:

logic, executed by the processor, for acquiring, through a first checking thread, a time point (T 1 ) at which a service node last fetches the service request from the service request queue;

logic, executed by the processor, for using, by the first checking thread, a time interval (D 1 ) between a current time point (N 1 ) and the time point (T 1 ) as a response time; and

logic, executed by the processor, for determining, by the first checking thread, that the response state of the service node is abnormal if the response time is greater than a first preset response time threshold.

19. The apparatus of claim 18 , the logic for monitoring response states further comprising:

logic, executed by the processor, for determining, through a second checking thread, that a storage unit of the service node includes a storage response timeout flag;

logic, executed by the processor, for using a time interval (D 2 ) between a flag-adding time point (T 2 ) corresponding to the storage response timeout flag and a current time point (N 2 ) as the response time; and

logic, executed by the processor, for determining that the response state of the service node is abnormal if the response time is greater than a second preset response time threshold.

20. The apparatus of claim 19 , wherein, prior to executing the logic for monitoring response states, the logic further comprises:

logic, executed by the processor, for acquiring, through a log recording thread, start and end times of log writing of the storage unit of the service node;

logic, executed by the processor, for using a time interval between the start and end times as a storage unit response time;

logic, executed by the processor, for adding the storage response timeout flag to the storage unit; and

logic, executed by the processor, for recording a time point (T 2 ) when the flag is added if the storage unit response time is greater than a third preset response time threshold.