Dependency resolution for lazy snapshots in storage cluster with delta log based architecture
Techniques for providing dependency resolution for lazy snapshots in a storage cluster with a delta log-based architecture. The techniques include creating a dependency relationship between a parent metadata page and a child metadata page, receiving one or more new delta updates of the parent metadata page, and writing, to a metadata delta log, the new delta updates of the parent metadata page as specialized delta update entries that preserve or maintain the new delta update values and their corresponding original (old) delta update values. By preserving or maintaining both the new delta update values and the corresponding old delta update values of the parent metadata page in the metadata delta log, constraints pertaining to the order of de-staging the delta updates of the parent and child metadata pages can be reduced.
1. A method of providing dependency resolution for a parent page and a child page in a storage cluster with a delta log-based architecture, comprising:
writing one or more new data values of a parent page to a data log, the new data values of the parent page having corresponding original data values, the new data values of the parent page being written to the data log as respective specialized data entries that maintain both the new data values and the corresponding original data values, and a dependency relationship existing between the parent page and a child page;
in a first de-staging operation for de-staging the parent page to data storage, building the parent page including the new data values maintained by the respective specialized data entries of the data log;
in a second de-staging operation for de-staging the child page to the data storage, building a base page including the original data values of the de-staged parent page maintained by the respective specialized data entries of the data log, and building the child page by applying one or more new data values of the child page to the base page; and
at a time of creation of the dependency relationship between the parent page and the child page, writing, to the data log, a first specialized data entry from among the respective specialized data entries that indicates the child page as being a child of the parent page.
2. The method of claim 1 further comprising:
writing one or more new data values of the child page to the data log after the first specialized data entry.
3. The method of claim 1 further comprising:
at the time of creation of the dependency relationship between the parent page and the child page, writing, to the data log, a second specialized data entry from among the respective specialized data entries that indicates the parent page as being a parent of the child page, the one or more new data values of the parent page being written to the data log after the second specialized data entry.
4. The method of claim 3 wherein the first de-staging operation includes locating the second specialized data entry in the data log, and applying, to the parent page, data values of the parent page that occur in the data log before and up to the second specialized data entry.
5. The method of claim 4 wherein the first de-staging operation further includes applying, to the parent page, the new data values of the parent page from the respective specialized data entries that occur in the data log after the second specialized data entry.
6. The method of claim 5 wherein the first de-staging operation further includes storing the parent page to data storage.
7. The method of claim 3 wherein the second de-staging operation includes determining whether or not the parent page has already been de-staged.
8. The method of claim 7 wherein the second de-staging operation further includes, having determined that the parent page has not yet been de-staged, obtaining a base page for the child page including locating the first specialized data entry in the data log, obtaining a logical index of the parent page from the first specialized data entry, and applying, to the base page, the data values of the parent page that occur in the data log before and up to the second specialized data entry.
9. The method of claim 8 wherein the second de-staging operation further includes, having obtained the base page, building the child page including applying, to the base page, one or more new data values of the child page written to the data log after the first specialized data entry.
10. The method of claim 7 wherein the second de-staging operation further includes, having determined that the parent page has already been de-staged, obtaining a base page for the child page including locating the first specialized data entry in the data log, obtaining a logical index of the parent page from the first specialized data entry, and loading a copy of the parent page from data storage.
11. The method of claim 10 wherein the second de-staging operation further includes applying, to the loaded copy of the parent page, the corresponding original data values of the parent page from the respective specialized data entries that occur in the data log after the second specialized data entry.
12. The method of claim 11 wherein the second de-staging operation further includes, having obtained the base page, building the child page including applying, to the base page, one or more new data values of the child page written to the data log after the first specialized data entry.
13. The method of claim 12 wherein the second de-staging operation further includes storing the child page to data storage.
14. The method of claim 1 wherein the writing of the one or more new data values of the parent page to the data log includes determining that the one or more corresponding original data values are unknown, and maintaining placeholder values for the one or more unknown original data values in the respective specialized data entries.
15. The method of claim 14 wherein the first de-staging operation includes loading a copy of the parent page from data storage, and replacing the placeholder values for the one or more unknown original data values in the respective specialized data entries with corresponding original data values from the loaded copy of the parent page.
16. A system for providing dependency resolution for a parent page and a child page in a storage cluster with a delta log-based architecture, the system comprising:
a memory; and
processing circuitry configured to execute program instructions out of the memory to:
write one or more new data values of a parent page to a data log,
wherein the new data values of the parent page have corresponding original data values,
wherein the new data values of the parent page are written to the data log as respective specialized data entries that maintain both the new data values and the corresponding original data values, and
wherein a dependency relationship exists between the parent page and a child page;
in a first de-staging operation for de-staging the parent page to data storage, build the parent page including the new data values maintained by the respective specialized data entries of the data log; and
in a second de-staging operation for de-staging the child page to the data storage, build a base page including the original data values of the de-staged parent page maintained by the respective specialized data entries of the data log, and build the child page by applying one or more new data values of the child page to the base page,
wherein the processing circuitry is further configured to execute the program instructions out of the memory to:
determine that the one or more corresponding original data values are unknown;
maintain placeholder values for the one or more unknown original data values in the respective specialized data entries;
load a copy of the parent page from data storage; and
replace the placeholder values for the one or more unknown original data values in the respective specialized data entries with corresponding original data values from the loaded copy of the parent page.
17. A non-transitory computer-readable storage medium encoded with program instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method of providing dependency resolution for a parent page and a child page in a storage cluster with a delta log-based architecture, the method comprising:
writing one or more new data values of a parent page to a data log, the new data values of the parent page having corresponding original data values, the new data values of the parent page being written to the data log as respective specialized data entries that maintain both the new data values and the corresponding original data values, and a dependency relationship existing between the parent page and a child page;
in a first de-staging operation for de-staging the parent page to data storage, building the parent page including the new data values maintained by the respective specialized data entries of the data log; and
in a second de-staging operation for de-staging the child page to the data storage, building a base page including the original data values of the de-staged parent page maintained by the respective specialized data entries of the data log, and building the child page by applying one or more new data values of the child page to the base page,
wherein the writing of the one or more new data values of the parent page to the data log includes determining that the one or more corresponding original data values are unknown, and maintaining placeholder values for the one or more unknown original data values in the respective specialized data entries; and
wherein the first de-staging operation includes loading a copy of the parent page from data storage, and replacing the placeholder values for the one or more unknown original data values in the respective specialized data entries with corresponding original data values from the loaded copy of the parent page.