MONITORING INPUT/OUTPUT AND PERSISTENT RESERVATION ACTIVITY PATTERNS TO DETECT DEGRADED PERFORMANCE OF A HIGH AVAILABILITY AND FAULT TOLERANT APPLICATION
A technique monitors input/output (I/O) and storage ownership takeover activity patterns to detect degraded performance of a highly available and fault tolerant application executing in a multi-site environment. Multiple instances of the application execute in different containers or pods running on virtual machines (VMs) of a compute layer within a containerized (e.g., Kubernetes) clustering configuration that extends across clusters of the sites. A storage layer of the clusters provides shared storage to the pods running on the VMs across the multiple sites. One of the sites is configured as an active storage site configured to receive and service I/O requests from the compute layer. A single instance of the application is active at a time and configured as a “compute owner” of the shared storage to issue the I/O requests to the shared storage. The compute owner and active storage site may not be co-located on the same site, leading to excessive I/O and storage ownership takeover activity patterns indicative of degraded performance. Upon detecting such patterns, the technique automatically triggers a storage failover to ensure that compute owner and active storage site are co-located at the same site.
1 . A non-transitory computer readable medium including program instructions for execution on a processor of a multi-cluster environment, the program instructions configured to:
monitor input/output (I/O) accesses and storage ownership takeover activity from a fault tolerant application executing on a container of a first cluster configured for failover to a second cluster of the environment, wherein the I/O accesses occur via a cluster storage interface driver executing in another container of the first cluster and wherein the I/O accesses are directed to shared storage replicated from the first cluster to the second cluster, and wherein the storage ownership takeover activity is directed to establishing and pre-empting ownership of the shared storage;
detect patterns of the I/O accesses and storage ownership activity from a container separate from that of container-based application is indicating degraded I/O performance to determine whether the fault tolerant container-based application and the shared storage are co-located at the first cluster; and
in response to determining that the fault tolerant application and shared storage are no longer co-located at the first cluster, trigger a failover of the shared storage to ensure co-location with the fault tolerant container-based application.
2 . The non-transitory computer readable medium of claim 1 , wherein the program instructions configured to monitor I/O accesses and storage ownership takeover activity includes program instructions configured to monitor removal and addition of storage connection pattern of activity to ensure that the failover of the fault tolerant container-based application and the shared storage are co-located at the first cluster.
3 . The non-transitory computer readable medium of claim 1 , wherein the program instructions configured to detect patterns of the I/O accesses and storage ownership takeover activity includes program instructions configured to determine whether the patterns exceed a threshold of pre-determined patterns of I/O activity.
4 . The non-transitory computer readable medium of claim 1 , wherein the program instructions configured to detect patterns of the I/O accesses and storage ownership takeover activity is based on a sliding window of the monitored I/O accesses.
5 . The non-transitory computer readable medium of claim 1 , wherein the fault tolerant container-based application is part of a container managed cluster having failover logic for the fault tolerant container-based application.
6 . The non-transitory computer readable medium of claim 1 , wherein during degraded I/O performance, a failover of the fault tolerant application to the second cluster proxies the I/O accesses to the shared storage at the first site.
7 . The non-transitory computer readable medium of claim 1 , wherein the detected patterns of I/O accesses and storage ownership takeover activity are configurable per fault tolerant container-based application.
8 . The non-transitory computer readable medium of claim 1 , wherein failover of the fault tolerant container-based application is uncoordinated with failover of the shared storage.
9 . The non-transitory computer readable medium of claim 1 , wherein the program instructions configured to detect patterns of I/O access and storage ownership takeover activity includes program instructions configured to gather statistics of an amount of data transferred during a sliding window time interval.
10 . The non-transitory computer readable medium of claim 1 , wherein an administratively initiated maintenance triggers failover of one of the fault tolerant container-based application or the shared storage such that they are no-longer co-located.
11 . A method comprising:
monitoring input/output (I/O) accesses from a fault tolerant container-based application executing on a first cluster of a multi-cluster environment configured for failover to a second cluster of the environment, wherein the I/O accesses occur via a cluster storage interface driver executing in another container of the first cluster and wherein the I/O accesses and storage ownership takeover activity are directed to shared storage replicated from the first cluster to the second cluster and wherein the storage ownership takeover activity is directed to establishing and pre-empting ownership of the shared storage;
detecting patterns of the I/O accesses and storage ownership takeover activity from a container separate from that of container-based application indicating degraded I/O performance to determine whether the fault tolerant application and the shared storage are co-located at the first cluster; and
in response to determining that the fault tolerant application and shared storage are no longer co-located at the first cluster, triggering a failover of the shared storage to ensure co-location with the fault tolerant container-based application.
12 . The method of claim 11 , wherein monitoring I/O accesses and storage ownership takeover activity comprises monitoring removal and addition of storage connection pattern of activity to ensure that the failover of the fault tolerant container-based application and the shared storage are co-located at the first cluster.
13 . The method of claim 11 , wherein detecting patterns of the I/O accesses and storage ownership takeover activity comprises determining whether the patterns exceed a threshold of pre-determined patterns of I/O activity.
14 . The method of claim 11 , wherein detecting patterns of the I/O accesses and storage ownership takeover activity is based on a sliding window of the monitored I/O accesses.
15 . The method of claim 11 , wherein the fault tolerant container-based application is part of a container managed cluster having failover logic for the fault tolerant container-based application.
16 . The method of claim 11 , wherein during degraded I/O performance, a failover of the fault tolerant application to the second site proxies the I/O accesses to the shared storage at the first cluster.
17 . The method of claim 11 , wherein the detected patterns of I/O accesses and storage ownership takeover activity are configurable per fault tolerant container-based application.
18 . The method of claim 11 , wherein failover of the fault tolerant container-based application is uncoordinated with failover of the shared storage.
19 . The method of claim 11 , wherein detecting patterns of I/O access and storage ownership takeover activity comprises gathering statistics of an amount of data transferred during a sliding window time interval.
20 . The method of claim 11 , wherein an administratively initiated maintenance triggers failover of one of the fault tolerant container-based application or the shared storage such that they are no-longer co-located.
21 . An apparatus comprising:
a network connecting a first node of nodes of a multi-site environment, the first node having a processor configured to execute program instructions configured to:
monitor input/output (I/O) accesses and storage ownership takeover activity from a fault tolerant application executing on a first site configured for failover to a second site of the environment, wherein the I/O accesses are directed to shared storage replicated from the first site to the second site, and wherein the storage ownership takeover activity is directed to establishing and pre-empting ownership of the shared storage;
detect patterns of the I/O accesses and storage ownership takeover activity indicating degraded I/O performance to determine whether the fault tolerant application and the shared storage are co-located at the first site; and
in response to determining that the fault tolerant application and shared storage are no longer co-located at the first site, trigger a failover of the shared storage to ensure co-location with the fault tolerant application.
22 . The apparatus of claim 21 , wherein the program instructions configured to monitor I/O accesses and storage ownership takeover activity are further configured to monitor removal and addition of storage connection pattern of activity to ensure that the failover of the fault tolerant application and the shared storage are co-located at the first site.
23 . The apparatus of claim 21 , wherein the program instructions configured to detect patterns of the I/O accesses and storage ownership takeover activity are further configured to determine whether the patterns exceed a threshold of pre-determined patterns of I/O activity.
24 . The apparatus of claim 21 , wherein the program instructions configured to detect patterns of the I/O accesses and storage ownership takeover activity is based on a sliding window of the monitored I/O accesses.
25 . The apparatus of claim 21 , wherein the fault tolerant application is part of a layered cluster having a compute witness for failover of the fault tolerant application, and wherein the determination that the fault tolerant application and shared storage are no longer co-located at the first site is independent of the compute witness.
26 . The apparatus of claim 21 , wherein during degraded I/O performance, a failover of the fault tolerant application to the second site proxies the I/O accesses to the shared storage at the first site.
27 . The apparatus of claim 21 , wherein the detected patterns of I/O accesses and storage ownership takeover activity are configurable per fault tolerant application.
28 . The apparatus of claim 21 , wherein failover of the fault tolerant application is uncoordinated with failover of the shared storage.
29 . The apparatus of claim 21 , wherein the program instructions configured to detect patterns of I/O access and storage ownership takeover activity are further configured to gather statistics of an amount of data transferred during a sliding window time interval.
30 . The apparatus of claim 21 , wherein an administratively initiated maintenance triggers failover of one of the fault tolerant application or the shared storage such that they are no-longer co-located.