Cache recovery method in a distributed storage system
An aspect of cache recovery includes transmitting entries of a write cache (WC) journal (“entries”) to all nodes and, for each node, recovering the entries, detecting entries with a logical address owned by the node, and performing a recovery operation. The operation includes for each entry, and upon determining the node owns the A2N slice: if the A2N slice has been continuously owned (CO) by the node, and the entry is not owned by the node, marking the entry as WC remote and entry updates are requested from a remote WC owner; if the A2N slice has not been CO by the node, and the entry is not owned by the node, maintaining the entry and continuing write flow operations, marking the entry as WC remote and all entry updates are requested from the remote WC owner and inserting the entry to a recovery list.
1. A method for cache recovery in a distributed storage system environment in response to a high availability (HA) event and redistribution of address-to-node (A2N) slices between currently alive nodes, the method comprising:
transmitting all entries of a write cache journal for each node to all the nodes; and
for each node:
recovering write cache journal entries and detecting all entries with a logical address owned by the node; and
performing a recovery operation, comprising:
for each write cache journal entry, and upon determining the node owns the A2N slice:
if the A2N slice has been continuously owned by the node, and the write cache journal entry is not owned by the node, marking the entry as write cache remote and all write cache journal updates are requested from a remote WC owner;
if the A2N slice has not been continuously owned by the node, and the write cache journal entry is not owned by the node, maintaining the entry and continuing write flow operations, marking the entry as WC remote and all journal updates are requested from the remote WC owner and inserting the entry to a write cache recovery list;
if the A2N slice has not been continuously owned by the node, and the write cache journal entry is owned by the node, inserting the entry to the write cache recovery list; and
if the A2N slice has been continuously owned by the node and the write cache entry is owned by the node, clearing a write cache remote flag in the write cache.
2. The method of claim 1 , wherein continuously owned indicates address ownership by a single node before and after the HA.
3. The method of claim 1 , wherein:
continuously owned address ownership of the A2N slice where the write cache journal entry is not owned by the node indicates the entry exists prior to the HA event but journal update ownership is not assigned to the node; and
non-continuous ownership of the A2N slice wherein the write cache journal entry is not owned by the node indicates the address owner has sole responsibility of the entry.
4. The method of claim 1 , wherein:
non-continuous ownership of the A2N slice where the write cache journal entry is owned by the node indicates both the A2H owner and the WC owner are responsible for the entry; and
continuous ownership of the A2N slice wherein the write cache journal entry is owned by the node indicates the entry existed prior to the HA event.
5. The method of claim 1 , further comprising:
recovering A2H entries of an A2H table for each node;
detecting addresses that point to a write cache journal entry;
marking the write cache journal entry to begin destage flow upon completion of recovery; and
removing the write cache journal entry from the write cache recovery list.
6. The method of claim 5 , further comprising:
upon detecting no address points to the write cache journal entry, continuing write flow operations upon completion of recovery.
7. The method of claim 1 , further comprising:
accessing write cache entries in the write cache recovery list indicating entries have no A2H reference; and
continuing write flow operations according to write cache entry state;
wherein entries having no A2H reference indicate entries in process of asynchronization where A2H is updated with a hash value of a page corresponding to address.
8. A system for cache recovery in a distributed storage system environment in response to a high availability (HA) event and redistribution of address-to-node (A2N) slices between currently alive nodes, the system comprising:
a memory comprising computer-executable instructions; and
a processor executing the computer-executable instructions, the computer-executable instructions when executed by the processor cause the processor to perform operations comprising:
transmitting all entries of a write cache journal for each node to all the nodes; and
for each node:
recovering write cache journal entries and detecting all entries with a logical address owned by the node; and
performing a recovery operation, comprising:
for each write cache journal entry, and upon determining the node owns the A2N slice:
if the A2N slice has been continuously owned by the node, and the write cache journal entry is not owned by the node, marking the entry as write cache remote and all write cache journal updates are requested from a remote WC owner;
if the A2N slice has not been continuously owned by the node, and the write cache journal entry is not owned by the node, maintaining the entry and continuing write flow operations, marking the entry as WC remote and all journal updates are requested from the remote WC owner and inserting the entry to a write cache recovery list;
if the A2N slice has not been continuously owned by the node, and the write cache journal entry is owned by the node, inserting the entry to the write cache recovery list; and
if the A2N slice has been continuously owned by the node and the write cache entry is owned by the node, clearing a write cache remote flag in the write cache.
9. The system of claim 8 , wherein continuously owned indicates address ownership by a single node before and after the HA.
10. The system of claim 8 , wherein:
continuous ownership of the A2N slice where the write cache journal entry is not owned by the node indicates the entry exists prior to the HA event but journal update ownership is not assigned to the node; and
non-continuous ownership of the A2N slice wherein the write cache journal entry is not owned by the node indicates the address owner has sole responsibility of the entry.
11. The system of claim 8 , wherein:
non-continuous ownership of the A2N slice where the write cache journal entry is owned by the node indicates both the A2H owner and the WC owner are responsible for the entry; and
continuous ownership of the A2N slice wherein the write cache journal entry is owned by the node indicates the entry existed prior to the HA event.
12. The system of claim 8 , wherein the operations further comprise:
recovering A2H entries of an A2H table for each node;
detecting addresses that point to a write cache journal entry;
marking the write cache journal entry to begin destage flow upon completion of recovery; and
removing the write cache journal entry from the write cache recovery list.
13. The system of claim 12 , wherein the operations further comprise:
upon detecting no address points to the write cache journal entry, continuing write flow operations upon completion of recovery.
14. The system of claim 8 , wherein the operations further comprise:
accessing write cache entries in the write cache recovery list indicating entries have no A2H reference; and
continuing write flow operations according to write cache entry state;
wherein entries having no A2H reference indicate entries in process of asynchronization where A2H is updated with a hash value of a page corresponding to address.
15. A computer program product embodied on a non-transitory computer readable medium, the computer program product including instructions that, when executed by a computer causes the computer to perform operations comprising:
transmitting all entries of a write cache journal for each node to all the nodes; and
for each node:
recovering write cache journal entries and detecting all entries with a logical address owned by the node; and
performing a recovery operation, comprising:
for each write cache journal entry, and upon determining the node owns the A2N slice:
if the A2N slice has been continuously owned by the node, and the write cache journal entry is not owned by the node, marking the entry as write cache remote and all write cache journal updates are requested from a remote WC owner;
if the A2N slice has not been continuously owned by the node, and the write cache journal entry is not owned by the node, maintaining the entry and continuing write flow operations, marking the entry as WC remote and all journal updates are requested from the remote WC owner and inserting the entry to a write cache recovery list;
if the A2N slice has not been continuously owned by the node, and the write cache journal entry is owned by the node, inserting the entry to the write cache recovery list; and
if the A2N slice has been continuously owned by the node and the write cache entry is owned by the node, clearing a write cache remote flag in the write cache.
16. The computer program product of claim 15 , wherein continuously owned indicates address ownership by a single node before and after the HA.
17. The computer program product of claim 15 , wherein:
continuous ownership of the A2N slice where the write cache journal entry is not owned by the node indicates the entry exists prior to the HA event but journal update ownership is not assigned to the node; and
non-continuous ownership of the A2N slice wherein the write cache journal entry is not owned by the node indicates the address owner has sole responsibility of the entry.
18. The computer program product of claim 15 , wherein:
non-continuous ownership of the A2N slice where the write cache journal entry is owned by the node indicates both the A2H owner and the WC owner are responsible for the entry; and
continuous ownership of the A2N slice wherein the write cache journal entry is owned by the node indicates the entry existed prior to the HA event.
19. The computer program product of claim 15 , wherein the operations further comprise:
recovering A2H entries of an A2H table for each node;
detecting addresses that point to a write cache journal entry;
marking the write cache journal entry to begin destage flow upon completion of recovery;
removing the write cache journal entry from the write cache recovery list; and
upon detecting no address points to the write cache journal entry, continuing write flow operations upon completion of recovery.
20. The computer program product of claim 15 , wherein the operations further comprise:
accessing write cache entries in the write cache recovery list indicating entries have no A2H reference; and
continuing write flow operations according to write cache entry state;
wherein entries having no A2H reference indicate entries in process of asynchronization where A2H is updated with a hash value of a page corresponding to address.