Error resistant write-ahead log
Embodiments are directed to error resistant logging. A write-ahead log (WAL) for preserving a verifiable record of file system activity may be provided such that the WAL may include a sequence of log blocks that each may include a cyclic redundancy check (CRC) value of a next adjacent log block in the WAL. In response to executing a file system operation associated with payload data, further actions may be performed, including: generating log blocks based on the payload data; generating a log segment that includes the log blocks; modifying a portion of the contents of a head-block of the log segment such that the modified head-block of the log segment has a CRC value that matches a CRC value of a tail-block of the WAL; copying the contents of the modified head-block of the log segment into the tail-block of the WAL.
1. A method for managing data in a file system over a network using one or more processors that execute instructions that are configured to cause actions, comprising:
generating a write-ahead log (WAL) for preserving a verifiable record of file system activity, wherein the WAL includes a sequence of log blocks that each include a cyclic redundancy check (CRC) value of a next adjacent log block in the WAL and a last log block in the WAL is a tail-block with a placeholder value for a CRC value of its next adjacent log block in the WAL;
in response to executing a file system operation associated with payload data, performing further actions, including:
generating one or more log blocks based on the payload data, wherein the one or more log blocks each include a portion of the payload data;
generating a log segment that includes the one or more log blocks, wherein each log block in the log segment includes a reference to a next adjacent log block in the log segment;
modifying a portion of the contents of a head-block of the log segment, wherein the modified head-block of the log segment has a CRC value that is modified to match a CRC value of the tail-block of the WAL, wherein the modification prevents one or more CRC errors occurring in the generation of the WAL for the file system activity;
copying the contents of the modified head-block of the log segment into the tail-block of the WAL; and
providing an acknowledgment that the file system operation is executed.
2. The method of claim 1 , wherein modifying the portion of the contents of the head-block of the log segment, further comprises:
determining a tweak data field in the head-block of the log segment;
determining a remainder portion of the head-block of the log segment absent the tweak data field;
determining a value for the tweak data field based on a combination of the determined value of the tweak data field in the head-block of the log segment and the remainder portion of the head-block of the log segment, wherein a CRC value for the combination of the determined value of the tweak data field of the head-block of the log segment and the remainder portion of the head-block of the log segment matches the CRC value of the tail-block of the WAL.
3. The method of claim 1 , further comprising:
determining a sequence of one or more remove log blocks in the WAL that are designated for removal from the WAL based on another file system operation;
determining a first log block in the WAL that is prior to and adjacent to the sequence of the one or more remove log blocks;
updating a reference in the first log block to point to a second log block in the WAL that is subsequent to and adjacent to the sequence of the one or more remove log blocks;
updating a reference to a next log block in the first log block to point to the second log block;
updating a next log block CRC field in the first log block to match a CRC field in the second log block; and
modifying a value of a tweak data field in the first log block to ensure a CRC field of the first block remains unchanged.
4. The method of claim 1 , wherein each log block in the WAL includes, a CRC value of the log block, a CRC value of its next adjacent log block, a tweak data field, and a payload data section.
5. The method of claim 1 , wherein copying the contents of the head-block of the log segment into the tail-block of the WAL, further comprises:
determining a location in a random access memory (RAM) that stores the tail-block of the WAL; and
replacing the contents of the tail-block of the WAL with the contents of the head-block of the log segment, wherein the location of the tail-block of the WAL in the RAM remains unchanged.
6. A system for managing data in a file system comprising:
a network computer, comprising:
a non-transitory memory device that stores at least instructions; and
one or more processors that execute the instructions that are configured to cause actions, including:
generating a write-ahead log (WAL) for preserving a verifiable record of file system activity, wherein the WAL includes a sequence of log blocks that each include a cyclic redundancy check (CRC) value of a next adjacent log block in the WAL and a last log block in the WAL is a tail-block with a placeholder value for a CRC value of its next adjacent log block in the WAL;
in response to executing a file system operation associated with payload data, performing further actions, including:
generating one or more log blocks based on the payload data, wherein the one or more log blocks each include a portion of the payload data;
generating a log segment that includes the one or more log blocks, wherein each log block in the log segment includes a reference to a next adjacent log block in the log segment;
modifying a portion of the contents of a head-block of the log segment, wherein the modified head-block of the log segment has a CRC value that is modified to match a CRC value of the tail-block of the WAL, wherein the modification prevents one or more CRC errors occurring in the generation of the WAL for the file system activity;
copying the contents of the modified head-block of the log segment into the tail-block of the WAL; and
providing an acknowledgment that the file system operation is executed; and
one or more client computers, comprising:
a non-transitory memory device that stores at least instructions; and
one or more processors that execute instructions that perform actions, including, providing one or more commands associated with the file system operation.
7. The system of claim 6 , wherein modifying the portion of the contents of the head-block of the log segment, further comprises:
determining a tweak data field in the head-block of the log segment;
determining a remainder portion of the head-block of the log segment absent the tweak data field;
determining a value for the tweak data field based on a combination of the determined value of the tweak data field in the head-block of the log segment and the remainder portion of the head-block of the log segment, wherein a CRC value for the combination of the determined value of the tweak data field of the head-block of the log segment and the remainder portion of the head-block of the log segment matches the CRC value of the tail-block of the WAL.
8. The system of claim 6 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:
determining a sequence of one or more remove log blocks in the WAL that are designated for removal from the WAL based on another file system operation;
determining a first log block in the WAL that is prior to and adjacent to the sequence of the one or more remove log blocks;
updating a reference in the first log block to point to a second log block in the WAL that is subsequent to and adjacent to the sequence of the one or more remove log blocks;
updating a reference to a next log block in the first log block to point to the second log block;
updating a next log block CRC field in the first log block to match a CRC field in the second log block; and
modifying a value of a tweak data field in the first log block to ensure a CRC field of the first block remains unchanged.
9. The system of claim 6 , wherein each log block in the WAL includes, a CRC value of the log block, a CRC value of its next adjacent log block, a tweak data field, and a payload data section.
10. The system of claim 6 , wherein copying the contents of the head-block of the log segment into the tail-block of the WAL, further comprises:
determining a location in a random access memory (RAM) that stores the tail-block of the WAL; and
replacing the contents of the tail-block of the WAL with the contents of the head-block of the log segment, wherein the location of the tail-block of the WAL in the RAM remains unchanged.
11. A processor readable non-transitory storage media that includes instructions for managing data in a file system over a network, wherein execution of the instructions, by one or more processors on one or more network computers, are configured to cause actions, comprising:
generating a write-ahead log (WAL) for preserving a verifiable record of file system activity, wherein the WAL includes a sequence of log blocks that each include a cyclic redundancy check (CRC) value of a next adjacent log block in the WAL and a last log block in the WAL is a tail-block with a placeholder value for a CRC value of its next adjacent log block in the WAL;
in response to executing a file system operation associated with payload data, performing further actions, including:
generating one or more log blocks based on the payload data, wherein the one or more log blocks each include a portion of the payload data;
generating a log segment that includes the one or more log blocks, wherein each log block in the log segment includes a reference to a next adjacent log block in the log segment;
modifying a portion of the contents of a head-block of the log segment, wherein the modified head-block of the log segment has a CRC value that is modified to match a CRC value of the tail-block of the WAL, wherein the modification prevents one or more CRC errors occurring in the generation of the WAL for the file system activity;
copying the contents of the modified head-block of the log segment into the tail-block of the WAL; and
providing an acknowledgment that the file system operation is executed.
12. The media of claim 11 , wherein modifying the portion of the contents of the head-block of the log segment, further comprises:
determining a tweak data field in the head-block of the log segment;
determining a remainder portion of the head-block of the log segment absent the tweak data field;
determining a value for the tweak data field based on a combination of the determined value of the tweak data field in the head-block of the log segment and the remainder portion of the head-block of the log segment, wherein a CRC value for the combination of the determined value of the tweak data field of the head-block of the log segment and the remainder portion of the head-block of the log segment matches the CRC value of the tail-block of the WAL.
13. The media of claim 11 , further comprising:
determining a sequence of one or more remove log blocks in the WAL that are designated for removal from the WAL based on another file system operation;
determining a first log block in the WAL that is prior to and adjacent to the sequence of the one or more remove log blocks;
updating a reference in the first log block to point to a second log block in the WAL that is subsequent to and adjacent to the sequence of the one or more remove log blocks;
updating a reference to a next log block in the first log block to point to the second log block;
updating a next log block CRC field in the first log block to match a CRC field in the second log block; and
modifying a value of a tweak data field in the first log block to ensure a CRC field of the first block remains unchanged.
14. The media of claim 11 , wherein each log block in the WAL includes, a CRC value of the log block, a CRC value of its next adjacent log block, a tweak data field, and a payload data section.
15. The media of claim 11 , wherein copying the contents of the head-block of the log segment into the tail-block of the WAL, further comprises:
determining a location in a random access memory (RAM) that stores the tail-block of the WAL; and
replacing the contents of the tail-block of the WAL with the contents of the head-block of the log segment, wherein the location of the tail-block of the WAL in the RAM remains unchanged.
16. A network computer for managing data in a file system, comprising:
a non-transitory memory device that stores at least instructions; and
one or more processors that execute the instructions that are configured to cause actions, including:
generating a write-ahead log (WAL) for preserving a verifiable record of file system activity, wherein the WAL includes a sequence of log blocks that each include a cyclic redundancy check (CRC) value of a next adjacent log block in the WAL and a last log block in the WAL is a tail-block with a placeholder value for a CRC value of its next adjacent log block in the WAL;
in response to executing a file system operation associated with payload data, performing further actions, including:
generating one or more log blocks based on the payload data, wherein the one or more log blocks each include a portion of the payload data;
generating a log segment that includes the one or more log blocks, wherein each log block in the log segment includes a reference to a next adjacent log block in the log segment;
modifying a portion of the contents of a head-block of the log segment, wherein the modified head-block of the log segment has a CRC value that is modified to match a CRC value of the tail-block of the WAL, wherein the modification prevents one or more CRC errors occurring in the generation of the WAL for the file system activity;
copying the contents of the modified head-block of the log segment into the tail-block of the WAL; and
providing an acknowledgment that the file system operation is executed.
17. The network computer of claim 16 , wherein modifying the portion of the contents of the head-block of the log segment, further comprises:
determining a tweak data field in the head-block of the log segment;
determining a remainder portion of the head-block of the log segment absent the tweak data field;
determining a value for the tweak data field based on a combination of the determined value of the tweak data field in the head-block of the log segment and the remainder portion of the head-block of the log segment, wherein a CRC value for the combination of the determined value of the tweak data field of the head-block of the log segment and the remainder portion of the head-block of the log segment matches the CRC value of the tail-block of the WAL.
18. The network computer of claim 16 , wherein the one or more processors execute instructions that perform actions, further comprising:
determining a sequence of one or more remove log blocks in the WAL that are designated for removal from the WAL based on another file system operation;
determining a first log block in the WAL that is prior to and adjacent to the sequence of the one or more remove log blocks;
updating a reference in the first log block to point to a second log block in the WAL that is subsequent to and adjacent to the sequence of the one or more remove log blocks;
updating a reference to a next log block in the first log block to point to the second log block;
updating a next log block CRC field in the first log block to match a CRC field in the second log block; and
modifying a value of a tweak data field in the first log block to ensure a CRC field of the first block remains unchanged.
19. The network computer of claim 16 , wherein each log block in the WAL includes, a CRC value of the log block, a CRC value of its next adjacent log block, a tweak data field, and a payload data section.
20. The network computer of claim 16 , wherein copying the contents of the head-block of the log segment into the tail-block of the WAL, further comprises:
determining a location in a random access memory (RAM) that stores the tail-block of the WAL; and
replacing the contents of the tail-block of the WAL with the contents of the head-block of the log segment, wherein the location of the tail-block of the WAL in the RAM remains unchanged.