Techniques for correcting errors in cached pages
A method of correcting errors in a data storage system including a first node, a second node, and shared persistent storage (the first and second nodes being configured to process data storage requests) is provided. The method includes (a) reading cached pages from a first cache disposed within the first node, the cached pages being cached versions of respective persistent pages stored in the shared persistent storage; (b) in response to determining that one of the cached pages is corrupted, requesting that the second node return to the first node a corresponding remote page from a second cache disposed within the second node, the cached page and the remote page each caching a same persistent page of the shared persistent storage; and (c) in response to determining that the remote page received from the second node by the first node is not corrupted, correcting the cached page using the remote page.
1. A method of correcting errors in a data storage system, the data storage system including a first node, a second node, and shared persistent storage, the first and second nodes configured to process data storage requests, the method comprising:
reading cached pages from a first cache disposed within the first node, the cached pages being cached versions of respective persistent pages stored in the shared persistent storage;
in response to determining that one of the cached pages is corrupted, requesting that the second node return to the first node a corresponding remote page from a second cache disposed within the second node, the corrupted cached page and the remote page each caching a same persistent page of the shared persistent storage;
in response to determining that the remote page received from the second node by the first node is not corrupted, correcting the corrupted cached page using the remote page;
in response to determining that another cached page from the first cache is corrupted, requesting that the second node return to the first node a corresponding other remote page from the second cache; and
in response to one of (a) the second cache not storing a corresponding other remote page as requested and (b) determining that the corresponding other remote page stored in the second cache is corrupted: correcting the other corrupted cached page without using the corresponding other remote page.
2. The method of claim 1 wherein the cached pages are metadata pages that store metadata regarding user data stored within the data storage system.
3. The method of claim 2 wherein correcting the other corrupted cached page without using the corresponding other remote page includes performing a metadata consistency check procedure (MCCP) to correct errors within the metadata by cross-checking all metadata pages stored within a metadata portion of the shared persistent storage.
4. The method of claim 3 wherein the method further comprises, prior to performing the MCCP:
requesting that the persistent shared storage return to the first node a corresponding persistent page that backs the other cached page; and
determining that the corresponding persistent page is also corrupted.
5. The method of claim 4 wherein the method further comprises, in response to requesting that the persistent shared storage return to the first node the corresponding persistent page, receiving, by the first node, the corresponding persistent page from a persistent write cache, the persistent write cache configured to store metadata to be written to the shared persistent storage prior to being flushed to the shared persistent storage.
6. The method of claim 2 wherein reading cached pages from the first cache is part of fulfilling one of (1) a user data READ operation and (2) a user data WRITE operation, wherein fulfilling the user data READ or WRITE operation includes reading metadata pages that store location information that allows the user data to be located.
7. The method of claim 6 wherein the data READ or WRITE operation is a user data WRITE operation, and wherein fulfilling the user data WRITE operation further includes performing a read-modify-write on metadata pages that currently store old location information that allows the user data to be located, including overwriting the old location information with new location information.
8. The method of claim 6 wherein the method further includes, prior to requesting that the second node send to the first node the corresponding other remote page from the second cache, upgrading a read lock on the other cached page to a read-modify-write lock.
9. The method of claim 1 wherein correcting the other corrupted cached page without using the corresponding other remote page includes:
requesting that the persistent shared storage return to the first node a corresponding persistent page that backs the other cached page; and
in response to determining that the corresponding persistent page received from the persistent shared storage by the first node is not corrupted, correcting the other cached page using the corresponding persistent page.
10. The method of claim 1 wherein correcting the other corrupted cached page without using the corresponding other remote page includes:
requesting that a third node of the data storage system send to the first node yet another corresponding remote page from a third cache disposed within the third node, the other corrupted cached page and the yet other corresponding remote page each caching a same persistent page stored in the persistent storage; and
in response to determining that the yet other corresponding remote page received from the third node by the first node is not corrupted, correcting the other cached page using the yet other corresponding remote page.
11. The method of claim 1 wherein determining that the remote page received from the second node by the first node is not corrupted includes:
receiving, by the first node, the remote page from the second node; and
determining, by the first node, that the remote page is not corrupted.
12. The method of claim 11 wherein determining that the remote page is not corrupted includes checking an error detecting code for internal inconsistency within the page.
13. The method of claim 12 wherein checking the error detecting code for internal inconsistency within the page includes applying a checksum.
14. The method of claim 12 wherein checking the error detecting code for internal inconsistency within the page includes applying a cyclic redundancy check.
15. A data storage apparatus comprising:
a first node including first processing circuitry coupled to memory and a first cache;
a second node including second processing circuitry coupled to memory and a second cache, the first and second nodes being configured to process data storage requests; and
shared persistent storage;
wherein the first processing circuitry coupled to memory is configured to cause the first node to:
read cached pages from the first cache, the cached pages being cached versions of respective persistent pages stored in the shared persistent storage;
in response to determining that one of the cached pages is corrupted, request that the second node return to the first node a corresponding remote page from the second cache, the corrupted cached page and the remote page each caching a same persistent page of the shared persistent storage;
in response to determining that the remote page as received from the second node by the first node is not corrupted, correct the corrupted cached page using the remote page;
in response to one of (a) the second cache not storing a corresponding remote page as requested and (b) a determination that the corresponding remote page stored in the second cache is corrupted: correct the corrupted cached page without using the corresponding remote page.
16. The data storage apparatus of claim 15 wherein the cached pages are metadata pages that store metadata regarding user data stored within the data storage apparatus.
17. The data storage apparatus of claim 15 wherein correcting the corrupted cached page without using the corresponding remote page includes:
requesting that the persistent shared storage return to the first node a corresponding persistent page that backs the corrupted cached page; and
in response to determining that the corresponding persistent page received from the persistent shared storage by the first node is not corrupted, correcting the corrupted cached page using the corresponding persistent page.
18. A computer program product comprising a non-transitory computer-readable storage medium storing a set of instructions, which, when performed by a computing device including a first node, a second node, and shared persistent storage, the first and second nodes being configured to process data storage requests, causes the computing device to correct errors by:
reading cached pages from a first cache disposed within the first node, the cached pages being cached versions of respective persistent pages stored in the shared persistent storage;
in response to determining that one of the cached pages is corrupted, requesting that the second node return to the first node a corresponding remote page from a second cache disposed within the second node, the corrupted cached page and the remote page each caching a same persistent page of the shared persistent storage;
in response to determining that the remote page received from the second node by the first node is not corrupted, correcting the corrupted cached page using the remote page;
in response to determining that another cached page from the first cache is corrupted, requesting that the second node return to the first node a corresponding other remote page from the second cache; and
in response to one of (a) the second cache not storing a corresponding other remote page as requested and (b) determining that the corresponding other remote page stored in the second cache is corrupted: correcting the other corrupted cached page without using the corresponding other remote page.