Distributed cache coherency protocol for reduced latency across WAN links
A share group for a distributed volume is separated into two non-overlapping, cluster-local SGCs (share group cliques). Each SGC has an independent local meta-directory master hosted by a computing node of the SGC cluster. Each SGC maintains an independent local directory for the distributed volume. The directory may be spread across multiple computing nodes of the SGC cluster. Changes to the distributed volume (writes) are coordinated across the pair of SGCs to provide data consistency. The coordination of these changes is conducted using relatively few inter-SGC round-trips. Read access to the distributed volume is safely conducted without inter-SGC coordination.
1. An apparatus comprising:
a first cluster comprising:
a first set of non-volatile back-end storage devices;
a plurality of first computing nodes which present a logical storage device and manage access to the first set of non-volatile back-end storage devices; and
a first local directory associated with the logical storage device, the first local directory indicating locations of data stored in cache of the first computing nodes;
wherein the first cluster is configured to operate as part of a storage system comprising a second cluster comprising a plurality of second computing nodes which present the logical storage device and manage access to a second set of non-volatile back-end storage devices, and a second local directory associated with the logical storage device, the second local directory indicating locations of data stored in cache of the second computing nodes.
2. The apparatus of claim 1 wherein the first local directory is hosted by one of the first computing nodes.
3. The apparatus of claim 1 wherein the first local directory is distributed across multiple ones of the first computing nodes.
4. The apparatus of claim 3 comprising a first local meta-directory hosted by one of the first computing nodes, the first local meta-directory indicating which local directory is associated with each extent of data of the logical storage device.
5. A data storage system comprising:
a first cluster comprising a plurality of first computing nodes which present a logical storage device and manage access to a first set of non-volatile back-end storage devices, and a first local directory associated with the logical storage device, the first local directory indicating locations of data stored in cache of the first computing nodes; and
a second cluster comprising a plurality of second computing nodes which present the logical storage device and manage access to a second set of non-volatile back-end storage devices, and a second local directory associated with the logical storage device, the second local directory indicating locations of data stored in cache of the second computing nodes;
wherein the first cluster is in communication with the second cluster via a network.
6. The data storage system of claim 5 wherein the first local directory is hosted by one of the first computing nodes and the second local directory is hosted by one of the second computing nodes.
7. The data storage system of claim 5 wherein the first local directory is distributed across multiple ones of the first computing nodes and the second local directory is distributed across multiple ones of the second computing nodes.
8. The data storage system of claim 7 further comprising a first local meta-directory hosted by one of the first computing nodes and a second local meta-directory hosted by one of the second computing nodes, the first and second local meta-directories indicating which of the first and second computing nodes, respectively, host portions of the local directory associated with each extent of data of the logical storage device.
9. The data storage system of claim 8 wherein, in response to a read request, a receiving computing node of the plurality of first computing nodes is configured to prompt the computing node which hosts the first local meta-directory to identify at least one of the computing nodes that hosts the first local directory for data associated with the read request.
10. The data storage system of claim 9 wherein the receiving computing node obtains a location of the data from the at least one of the computing nodes that hosts the first local directory for data associated with the read request, and uses that location to obtain a copy of the data.
11. The data storage system of claim 8 wherein, in response to a write request, a receiving computing node of the plurality of first computing nodes is configured to prompt the computing node which hosts the first local meta-directory to contact at least one of the computing nodes that hosts the first local directory for data associated with the write request to obtain locks and maintain cache coherency.
12. The data storage system of claim 11 wherein the computing node which hosts the first local meta-directory contacts the computing node which hosts the second local meta-directory to prompt the computing node which hosts the second local meta-directory to contact at least one of the computing nodes that hosts the second local directory for data associated with the write request to obtain locks and maintain cache coherency.
13. The data storage system of claim 12 wherein, in response to an overlapping write request, the computing node which hosts the first local meta-directory and the computing node which hosts the second local meta-directory independently determine priority of the write request relative to the overlapping write request and execute the write requests in order as determined by priority.
14. The data storage system of claim 13 wherein a lock obtained for a lower priority write is released to an overlapping higher priority write.
15. A method comprising:
in a first cluster comprising a plurality of first computing nodes:
presenting a logical storage device backed by a first set of non-volatile back-end storage devices;
maintaining a first local directory associated with the logical storage device, the first local directory being distributed across multiple ones of the first computing nodes and indicating locations of data stored in cache of the first computing nodes; and
maintaining a first local meta-directory hosted by one of the first computing nodes, the first local meta-directory indicating which first computing nodes host portions of the local directory associated with each extent of data of the logical storage device; and
in a second cluster comprising a plurality of second computing nodes:
presenting the logical storage device backed by a second set of non-volatile back-end storage devices;
maintaining a second local directory associated with the logical storage device, the second local directory being distributed across multiple ones of the second computing nodes and indicating locations of data stored in cache of the second computing nodes; and
maintaining a second local meta-directory hosted by one of the second computing nodes, the second local meta-directory indicating which second computing nodes host portions of the local directory associated with each extent of data of the logical storage device.
16. The method of claim 15 comprising, in response to a read request, a receiving computing node of the plurality of first computing nodes prompting the computing node which hosts the first local meta-directory to identify at least one of the computing nodes that hosts the first local directory for data associated with the read request.
17. The method of claim 16 comprising the receiving computing node obtaining a location of the data from the at least one of the computing nodes that hosts the first local directory for data associated with the read request, and using that location to obtain a copy of the data.
18. The method of claim 15 comprising, in response to a write request, a receiving computing node of the plurality of first computing nodes prompting the computing node which hosts the first local meta-directory to contact at least one of the computing nodes that hosts the first local directory for data associated with the write request to obtain locks and maintain cache coherency.
19. The method of claim 18 comprising the computing node which hosts the first local meta-directory contacting the computing node which hosts the second local meta-directory to prompt the computing node which hosts the second local meta-directory to contact at least one of the computing nodes that hosts the second local directory for data associated with the write request to obtain locks and maintain cache coherency.
20. The method of claim 19 comprising, in response to an overlapping write request, the computing node which hosts the first local meta-directory and the computing node which hosts the second local meta-directory independently determining priority of the write request relative to the overlapping write request and executing the write requests in order as determined by priority.