Verifying the consistency of slice allocation metadata
A technique for verifying the consistency of slice allocation metadata includes accessing, from user space of an operating system running on the data storage apparatus, a set of drivers running in kernel space of the operating system to obtain slice allocation metadata from the set of drivers, and identifying discrepancies in slice allocation metadata returned from the set of drivers on a per-file-system basis.
1. A method of verifying the consistency of slice allocation metadata in a data storage apparatus, the method comprising:
accessing, from userspace of an operating system, a set of drivers running in kernel space of the operating system to obtain a first set of slice allocation metadata, the first set of slice allocation metadata describing, for each of a first set of storage slices of the data storage apparatus, an identifier of any file system to which the slice has been allocated;
accessing, from userspace of the operating system, the set of drivers running in kernel space of the operating system to obtain a second set of slice allocation metadata, the second set of slice allocation metadata describing, for each of a second set of storage slices of the data storage apparatus, an identifier of a file system to which the slice has been allocated; and
for each file system to which the storage slices described in the first set of slice allocation metadata have been allocated, (1) identifying discrepancies between (a) slice allocation metadata identifying the file system in the first set of slice allocation metadata and (b) slice allocation metadata identifying the same file system in the second set of slice allocation metadata and (2) storing diagnostic information describing the discrepancies in a diagnostic file.
2. The method of claim 1 ,
wherein the data storage apparatus stores multiple file systems,
wherein each of the file systems is built upon a respective sparse volume,
wherein the data storage apparatus includes a first driver that tracks allocations of slices from a storage pool of the data storage apparatus to the file systems, and
wherein the data storage apparatus includes a second driver that tracks arrangements of slices within the sparse volumes.
3. The method of claim 2 , wherein the data storage apparatus includes a first storage processor and a second storage processor, and wherein accessing the set of drivers to obtain the first set of slice allocation metadata includes merging slice allocation metadata obtained from the first storage processor with slice allocation metadata obtained from the second storage processor to generate the first set of slice allocation metadata.
4. The method of claim 3 , further comprising:
identifying a slice allocation conflict when merging slice allocation metadata from the first storage processor with slice allocation metadata from the second storage processor, the slice allocation conflict indicating that a slice is allocated to one file system according to the slice allocation metadata obtained from the first storage processor but that the same slice is allocated to another file system according to the slice allocation metadata obtained from the second storage processor; and
generating an error message in response to identifying the slice allocation conflict.
5. The method of claim 2 , further comprising distributing the first set of slice allocation metadata among a set of output files in user space such that each output file is specific to a respective file system and is made to include slice allocation metadata for slices allocated to the respective file system.
6. The method of claim 5 , further comprising:
identifying a set of free slices described in the first set of slice allocation metadata, wherein free slices are storage slices that the first set of slice allocation metadata describes as not being allocated to any file system; and
storing slice allocation metadata for all such free slices in another output file.
7. The method of claim 6 , wherein storing diagnostic information describing the discrepancies in a diagnostic file includes storing diagnostic information for different file systems in respective diagnostic files, one for each file system.
8. The method of claim 7 , further comprising storing diagnostic information for the set of free slices in another diagnostic file.
9. The method of claim 8 , further comprising generating summary output that provides aggregated information about corrupted file systems, including kinds of corruption found in each corrupted file system and a number of occurrences of each kind of corruption found in each corrupted file system.
10. The method of claim 2 , further comprising generating summary output that identifies slice allocation metadata inconsistencies between different file systems.
11. The method of claim 2 , further comprising:
accessing, from userspace of the operating system, the set of drivers running in kernel space of the operating system to obtain a third set of slice allocation metadata, the third set of slice allocation metadata describing, for each of a third set of storage slices of the data storage apparatus, an identifier of a file system to which the slice has been allocated,
wherein the data storage apparatus includes a third driver that tracks allocations of slices in the file systems of the data storage apparatus, and
wherein identifying discrepancies operates to identify discrepancies between the first, second, and third sets of slice allocation metadata.
12. The method of claim 11 , wherein identifying discrepancies and storing diagnostic information are performed in response to receiving a command to run a program in user space, the command including an identifier of a particular storage pool for which verification of slice allocation metadata consistency is to be performed.
13. The method of claim 12 , further comprising, after storing diagnostic information, running a recovery tool in the kernel of the operating system to repair slice allocation metadata errors.
14. A computerized apparatus, comprising:
a set of processors; and
memory, coupled to the set of processors, the memory storing executable instructions, which when executed by the set of processors cause the set of processors to perform a method of verifying the consistency of slice allocation metadata in a data storage apparatus, the method comprising:
accessing, from userspace of an operating system, a set of drivers running in kernel space of the operating system to obtain a first set of slice allocation metadata, the first set of slice allocation metadata describing, for each of a first set of storage slices of the data storage apparatus, an identifier of any file system to which the slice has been allocated;
accessing, from userspace of the operating system, the set of drivers running in kernel space of the operating system to obtain a second set of slice allocation metadata, the second set of slice allocation metadata describing, for each of a second set of storage slices of the data storage apparatus, an identifier of a file system to which the slice has been allocated; and
for each file system to which the storage slices described in the first set of slice allocation metadata have been allocated, (1) identifying discrepancies between (a) slice allocation metadata identifying the file system in the first set of slice allocation metadata and (b) slice allocation metadata identifying the same file system in the second set of slice allocation metadata and (2) storing diagnostic information describing the discrepancies in a diagnostic file.
15. The computerized apparatus of claim 14 ,
wherein the data storage apparatus stores multiple file systems,
wherein each of the file systems is built upon a respective sparse volume,
wherein the data storage apparatus includes a first driver that tracks allocations of slices from a storage pool of the data storage apparatus to the file systems, and
wherein the data storage apparatus includes a second driver that tracks arrangements of slices within the sparse volumes.
16. The computerized apparatus of claim 15 , wherein the data storage apparatus includes a first storage processor and a second storage processor, and wherein accessing the set of drivers to obtain the first set of slice allocation metadata includes merging slice allocation metadata obtained from the first storage processor with slice allocation metadata obtained from the second storage processor to generate the first set of slice allocation metadata.
17. A non-transitory computer-readable medium including instructions which, when executed by a set of processors of a computing device, cause the set of processors to perform a method of verifying the consistency of slice allocation metadata in a data storage apparatus, the method comprising:
accessing, from userspace of an operating system, a set of drivers running in kernel space of the operating system to obtain a first set of slice allocation metadata, the first set of slice allocation metadata describing, for each of a first set of storage slices of the data storage apparatus, an identifier of any file system to which the slice has been allocated;
accessing, from userspace of the operating system, the set of drivers running in kernel space of the operating system to obtain a second set of slice allocation metadata, the second set of slice allocation metadata describing, for each of a second set of storage slices of the data storage apparatus, an identifier of a file system to which the slice has been allocated; and
for each file system to which the storage slices described in the first set of slice allocation metadata have been allocated, (1) identifying discrepancies between (a) slice allocation metadata identifying the file system in the first set of slice allocation metadata and (b) slice allocation metadata identifying the same file system in the second set of slice allocation metadata and (2) storing diagnostic information describing the discrepancies in a diagnostic file.
18. The non-transitory computer-readable medium of claim 17 , wherein the method further comprises generating summary output that stores aggregated information about corrupted file systems, including kinds of corruption found in each corrupted file system and a number of occurrences of each kind of corruption found in each corrupted file system.
19. The non-transitory computer-readable medium of claim 17 , wherein the method further comprises generating summary output that identifies slice allocation metadata inconsistencies between different file systems.
20. The non-transitory computer-readable medium of claim 17 , wherein the method further comprises distributing the first set of slice allocation metadata among a set of output files in user space such that each output file is specific to a respective file system and is made to include slice allocation metadata for slices allocated to the respective file system.