System and method of handling journal space in a storage cluster with multiple delta log instances
Techniques for handling journal space in a storage cluster with multiple delta log instances. The techniques include writing delta updates for a respective metadata type to an “active” set of data containers in a delta log instance and raw delta updates to a raw delta log, switching a designation of the “active” set of data containers from “active” to “de-staging” once one or more of the “active” set of data containers has been filled, writing a bookmark for the respective metadata type to the raw delta log and a bookmark list, determining that a de-staging operation has been completed for writing the delta updates from the “de-staging” set of data containers to a storage array, determining that the bookmark for the respective metadata type is the oldest bookmark in the list, and reclaiming space between a tail of the raw delta log and the bookmark written to the raw delta log.
1. A method of handling journal space in a storage cluster with multiple delta log instances, comprising:
writing delta updates of a first metadata type to both a respective set of data containers in a first delta log instance and a raw delta log;
in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, writing a first bookmark for the first metadata type to both the raw delta log and a bookmark list; and
in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array:
if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaiming at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and
if the first bookmark is determined not to be the oldest bookmark in the bookmark list, deferring the reclaiming of the first space in the raw delta log.
2. The method of claim 1 further comprising:
in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, removing the first bookmark for the first metadata type from the bookmark list.
3. The method of claim 1 further comprising:
writing delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log; and
in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, writing a second bookmark for the second metadata type to both the raw delta log and the bookmark list.
4. The method of claim 3 further comprising:
in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array:
determining that the second bookmark is not the oldest bookmark in the bookmark list; and
deferring reclaiming second space in the raw delta log.
5. The method of claim 4 further comprising:
in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array:
determining that the first bookmark is the oldest bookmark in the bookmark list; and
reclaiming at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log.
6. The method of claim 1 further comprising:
setting a threshold specifying a minimum allowed free space in the raw delta log.
7. The method of claim 6 further comprising:
determining that free space in the raw delta log is below the set threshold; and
forcing at least the first de-staging operation to be performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array.
8. A storage node of a storage cluster, comprising:
a volatile memory including a first delta log instance and a bookmark list, the first delta log instance having a respective set of data containers;
a persistent memory including a raw delta log; and
processing circuitry configured to execute program instructions to:
write delta updates of a first metadata type to both the respective set of data containers in the first delta log instance and the raw delta log;
in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, write a first bookmark for the first metadata type to both the raw delta log and the bookmark list; and
in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array:
if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaim at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and
if the first bookmark is determined not to be the oldest bookmark in the bookmark list, defer the reclaiming of the first space in the raw delta log.
9. The storage node of claim 8 wherein the processing circuitry is further configured to execute the program instructions, in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, to remove the first bookmark for the first metadata type from the bookmark list.
10. The storage node of claim 8 wherein the processing circuitry is further configured to execute the program instructions to:
write delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log; and
in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, write a second bookmark for the second metadata type to both the raw delta log and the bookmark list.
11. The storage node of claim 10 wherein the processing circuitry is further configured to execute the program instructions to:
in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array:
determine that the second bookmark is not the oldest bookmark in the bookmark list; and
defer reclaiming second space in the raw delta log.
12. The storage node of claim 11 wherein the processing circuitry is further configured to execute the program instructions to:
in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array:
determine that the first bookmark is the oldest bookmark in the bookmark list; and
reclaim at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log.
13. The storage node of claim 8 wherein the raw delta log is configured as a ring buffer having a head and a tail, and wherein the processing circuitry is further configured to execute the program instructions to write the delta updates of the first metadata type at the head of the raw delta log.
14. The storage node of claim 13 wherein the processing circuitry is further configured to execute the program instructions to write the first bookmark for the first metadata type at the head of the raw delta log.
15. The storage node of claim 8 wherein the bookmark list is configured as a linked list having a head and a tail, and wherein the processing circuitry is further configured to execute the program instructions to write the first bookmark for the first metadata type at the head of the linked list.
16. The storage node of claim 8 wherein the processing circuitry is further configured to execute the program instructions to set a threshold specifying a minimum allowed free space in the raw delta log.
17. The storage node of claim 16 wherein the processing circuitry is further configured to execute the program instructions to determine that free space in the raw delta log is below the set threshold, and force at least the first de-staging operation to be performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array.
18. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising:
writing delta updates of a first metadata type to both a respective set of data containers in a first delta log instance and a raw delta log;
in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, writing a first bookmark for the first metadata type to both the raw delta log and a bookmark list; and
in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array:
if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaiming at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and
if the first bookmark is determined not to be the oldest bookmark in the bookmark list, deferring the reclaiming of the first space in the raw delta log.
19. The computer program product of claim 18 wherein the method further comprises:
in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, removing the first bookmark for the first metadata type from the bookmark list.
20. The computer program product of claim 18 wherein the method further comprises:
writing delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log;
in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, writing a second bookmark for the second metadata type to both the raw delta log and the bookmark list;
in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array:
determining that the second bookmark is not the oldest bookmark in the bookmark list; and
deferring reclaiming second space in the raw delta log; and
in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array:
determining that the first bookmark is the oldest bookmark in the bookmark list; and
reclaiming at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log.