Data storage system with processor scheduling using distributed peek-poller threads
A method of multitasked operation of a data storage node includes executing provider threads of an application, the provider threads distributed among processing cores along with respective peek-poller threads, each provider thread having regular operations of (1) continually polling interfaces of the data storage node for application events requiring processing, and (2) performing processing tasks for the application events until there are no unprocessed additional events for the provider thread, and thereupon entering a sleep state. The method further includes regularly executing the peek-poller threads on the processing cores, each peek-poller thread having operations of (1) checking interfaces of the data storage node for new application events to be processed by respective provider threads currently in the sleep state, and (2) for each new application event, activating the respective provider thread on the respective processing core.
1. A method of multitasked operation of a data storage node of a data storage system, comprising:
executing provider threads of an application, the provider threads being distributed among corresponding processing cores of the data storage node along with respective peek-poller threads, each provider thread having regular operations upon becoming activated on a respective processing core of (1) continually polling respective interfaces of the data storage node for application events requiring processing, and (2) performing processing tasks for the application events until there are no unprocessed additional events for the provider thread, and thereupon entering a sleep state; and
regularly executing the peek-poller threads on the respective processing cores, each peek-poller thread having operations of (1) checking interfaces of the data storage node for new application events to be processed by respective provider threads currently in the sleep state, and (2) for each new application event, activating the respective provider thread on the respective processing core.
2. The method of claim 1 , wherein:
the application is a block-storage application and the provider threads are block threads each including a respective set of IO providers for performing the polling of the respective interfaces; and
the block-storage application includes respective inner schedulers executed on the processing cores, each including a respective peek-poller thread.
3. The method of claim 2 , wherein the interfaces of the data storage node include (1) a device interface to a data storage medium of the data storage node structured as logical storage devices, (2) a host network interface to a host network and host computers connected thereto, and (3) a cluster network interface to a cluster network interconnecting the data storage node with other data storage nodes of the data storage system, and wherein the set of IO providers includes (a) a front-end IO provider operative to poll the host network interface for new IO requests, (b) an RPC messaging IO provider operative to poll the cluster network interface for new messages from other data storage nodes 16 in the system, (c) an RDMA messaging IO provider operative to poll the cluster network interface for completion of remote DMA (RDMA) transfers, and (d) a back-end IO provider operative to poll the device interface for completion of IO requests initiated by the block thread to storage devices of the data storage node.
4. The method of claim 1 , wherein the application is a first application, and further including, by a system scheduler, observing scheduling rules including (1) that a provider thread that is executing does not get preempted even when a thread of a second application is ready to run, and (2) that a thread of the second application that is executing is preempted by a waiting provider thread that has become ready to run after being awakened.
5. The method of claim 4 , wherein a scheduling policy for the first application is configured with a priority higher than a priority of the second application, ensuring that the scheduling allows a ready provider thread to run until it voluntarily releases the respective processing core.
6. The method of claim 1 , wherein entering the sleep state includes executing a release decision that checks if all in-flight operations of the block thread are currently in a waiting state to decide to release the processing core.
7. The method of claim 6 , wherein the release decision also takes into account a continuous running duration of the block thread, and if the running duration exceeds a predetermined threshold, then releasing the processing core even if not all in-flight operations are in waiting state, to prevent starvation if another application.
8. The method of claim 6 , wherein the release decision further checks whether there is at least one other block thread not suspended as a condition to releasing the processing core, to avoid a deadlock situation in which there is no running block thread that can execute a peek-poller thread to wake up any block threads.
9. The method of claim 1 , further including, as a condition to executing the peek-poller threads, checking whether a configurable setting has a value that enables processor core sharing, and executing the peek-poller threads only when the configurable setting has the sharing-enabling value.
10. The method of claim 1 , further including regularly executing a keeper peek-poller thread that monitors for an unusually long delay in activating at least one peek-poller thread, and conditionally performing the checking and activating when none of the peek-poller threads has performed the checking and activating for a long delay time.
11. A data storage node for use in a data storage system, comprising:
data storage devices for persistent secondary storage of data;
a set of interfaces including at least a front-end interface to a host computer network and a device interface to the data storage devices; and
a storage processor including a plurality of processing cores, the storage processor executing computer program instructions of an application, the execution of the computer program instructions causing multitasked operation of the data storage node including:
executing provider threads of an application, the provider threads being distributed among corresponding processing cores of the data storage node along with respective peek-poller threads, each provider thread having regular operations upon becoming activated on a respective processing core of (1) continually polling respective interfaces of the data storage node for application events requiring processing, and (2) performing processing tasks for the application events until there are no unprocessed additional events for the provider thread, and thereupon entering a sleep state; and
regularly executing the peek-poller threads on the respective processing cores, each peek-poller thread having operations of (1) checking interfaces of the data storage node for new application events to be processed by respective provider threads currently in the sleep state, and (2) for each new application event, activating the respective provider thread on the respective processing core.
12. The data storage node of claim 11 , wherein:
the application is a block-storage application and the provider threads are block threads each including a respective set of IO providers for performing the polling of the respective interfaces; and
the block-storage application includes respective inner schedulers executed on the processing cores, each including a respective peek-poller thread.
13. The data storage node of claim 12 , wherein the interfaces of the data storage node include (1) a device interface to a data storage medium of the data storage node structured as logical storage devices, (2) a host network interface to a host network and host computers connected thereto, and (3) a cluster network interface to a cluster network interconnecting the data storage node with other data storage nodes of the data storage system, and wherein the set of IO providers includes (a) a front-end IO provider operative to poll the host network interface for new IO requests, (b) an RPC messaging IO provider operative to poll the cluster network interface for new messages from other data storage nodes 16 in the system, (c) an RDMA messaging IO provider operative to poll the cluster network interface for completion of remote DMA (RDMA) transfers, and (d) a back-end IO provider operative to poll the device interface for completion of IO requests initiated by the block thread to storage devices of the data storage node.
14. The data storage node of claim 11 , wherein the application is a first application, and further including, by a system scheduler, observing scheduling rules including (1) that a provider thread that is executing does not get preempted even when a thread of a second application is ready to run, and (2) that a thread of the second application that is executing is preempted by a waiting provider thread that has become ready to run after being awakened.
15. The data storage node of claim 14 , wherein a scheduling policy for the first application is configured with a priority higher than a priority of the second application, ensuring that the scheduling allows a ready provider thread to run until it voluntarily releases the respective processing core.
16. The data storage node of claim 11 , wherein entering the sleep state includes executing a release decision that checks if all in-flight operations of the block thread are currently in a waiting state to decide to release the processing core.
17. The data storage node of claim 16 , wherein the release decision also takes into account a continuous running duration of the block thread, and if the running duration exceeds a predetermined threshold, then releasing the processing core even if not all in-flight operations are in waiting state, to prevent starvation if another application.
18. The data storage node of claim 16 , wherein the release decision further checks whether there is at least one other block thread not suspended as a condition to releasing the processing core, to avoid a deadlock situation in which there is no running block thread that can execute a peek-poller thread to wake up any block threads.
19. The data storage node of claim 11 , wherein the multitasked operation further includes, as a condition to executing the peek-poller threads, checking whether a configurable setting has a value that enables processor core sharing, and executing the peek-poller threads only when the configurable setting has the sharing-enabling value.
20. The data storage node of claim 11 , wherein the multitasked operation further includes regularly executing a keeper peek-poller thread that monitors for an unusually long delay in activating at least one peek-poller thread, and conditionally performing the checking and activating when none of the peek-poller threads has performed the checking and activating for a long delay time.