Computer data file system with consistency groups as basic file system objects
A computer has a secondary storage device with a filesystem for persistent storage of application data, the filesystem having filesystem objects including files, directories, and group objects such as consistency groups. The group objects specify respective groups of the filesystem objects on which certain filesystem group operations act as a unit. The filesystem also has filesystem routines for performing filesystem operations including single-object operations and group operations. A single-object operation acts upon a specified user file or directory independently (e.g., opening or reading from a file), while a group operation acts upon a respective set of filesystem objects grouped together by a specified group object.
1. A computer having a secondary storage device for persistent storage of application data generated by a data storage application, the secondary storage device having a filesystem thereon, the filesystem including:
filesystem objects including files, directories, and consistency group objects, the files storing the application data, the directories functioning as branches in a tree graph interpretation of an organization of the filesystem objects, and the consistency group objects specifying respective groups of the filesystem objects on which certain filesystem group operations act; and
filesystem routines for performing filesystem operations including single-object operations and consistency group operations, a single-object operation acting upon a specified user file or directory independently, a consistency group operation acting upon a respective consistency group of filesystem objects grouped together by a specified consistency group object,
wherein the consistency group operations include group snapshot operations each specifying a respective consistency group object and each including (i) first transactionally suspending IO to all the objects of the respective consistency group and (ii) then creating a snapset object containing a single point in time snapshot of all the objects of the respective consistency group, the group snapshot operations being carried out by the file system without special programming of the data storage application.
2. A method of operating a computer having a secondary storage device for persistent storage of application data generated by a data storage application, comprising:
storing the application data in a filesystem on the secondary storage device, the filesystem including filesystem objects including files, directories, and consistency group objects, the files storing the application data, the directories functioning as branches in a tree graph interpretation of an organization of the filesystem objects, and the consistency group objects specifying respective groups of the filesystem objects on which certain filesystem group operations act; and
executing filesystem routines for performing filesystem operations including single-object operations and consistency group operations, a single-object operation acting upon a specified user file or directory independently, a consistency group operation acting upon a respective consistency group of filesystem objects grouped together by a specified consistency group object,
wherein the consistency group operations include group snapshot operations each specifying a respective consistency group object and each including (i) first transactionally suspending IO to all the objects of the respective consistency group and (ii) then creating a snapset object containing a single point in time snapshot of all the objects of the respective consistency group, the group snapshot operations being carried out by the file system without special programming of the data storage application.
3. The computer of claim 1 , having a block subsystem including block components and filesystem components of the filesystem, the block subsystem being responsible for providing block-oriented storage resources including logical volumes to applications of the computer using the secondary storage device, the block components defining and operating upon logical storage objects stored as the files in the filesystem, the filesystem providing basic operations including reading and writing as well as higher-level operations including the group snapshot operations.
4. The computer of claim 1 , wherein the filesystem includes components at three functional layers including a namespace layer, a mapper layer, and a cache layer, the namespace layer performing higher-level functions including managing the filesystem objects as single logical extents, the mapper layer performing more detailed underlying operations including maintaining an indirect-block tree of inodes as filesystem metadata identifying underlying storage blocks for the filesystem objects, the cache layer being a transaction layer enabling the namespace layer to start transactions and have them completed automatically, and to buffer intermediate results so that either an entire transaction is committed or it is aborted.
5. The computer of claim 1 , wherein the group snapshot operation either completes for all filesystem objects of a respective consistency group or does not complete for any filesystem object of the respective consistency group.
6. The computer of claim 5 , wherein the consistency group objects are used in connection with storage operations including one or more of replication, rebuilding, or migration.
7. The computer of claim 6 , wherein the consistency group objects are used in connection with restoring sets of disks across multiple virtual machines collectively requiring restoration of the same restored image after a disruption.
8. The computer of claim 1 , wherein each of the consistency group objects is layered on top of a respective directory and is formed as a super-type of directory that incorporates functionality of underlying directory objects.
9. The computer of claim 1 , wherein the consistency group objects are distinct objects each having structure and functionality similar to corresponding structure and functionality of the directories but not utilizing or relying on the directories for the structure and functionality.
10. The computer of claim 1 , wherein the consistency group operations performed on the consistency group objects include CreateObject, DeleteObject, CreateReferenceLink, DeleteReferenceLink, CreateReplica, EnumerateObjects, and EnumerateSnapsets, wherein a reference link is a reference entry linking two shared objects such that an update to either object is visible on both objects.
11. The method of claim 2 , having a block subsystem including block components and filesystem components of the filesystem, the block subsystem being responsible for providing block-oriented storage resources including logical volumes to applications of the computer using the secondary storage device, the block components defining and operating upon logical storage objects stored as the files in the filesystem, the filesystem providing basic operations including reading and writing as well as higher-level operations including the group snapshot operations.
12. The method of claim 2 , wherein the filesystem includes components at three functional layers including a namespace layer, a mapper layer, and a cache layer, the namespace layer performing higher-level functions including managing filesystem objects as single logical extents, the mapper layer performing more detailed underlying operations including maintaining an indirect-block tree of inodes as filesystem metadata identifying underlying storage blocks for the filesystem objects, the cache layer being a transaction layer enabling the namespace layer to start transactions and have them completed automatically, and to buffer intermediate results so that either an entire transaction is committed or it is aborted.
13. The method of claim 2 , wherein the group snapshot operation either completes for all filesystem objects of a respective consistency group or does not complete for any filesystem object of the respective consistency group.
14. The method of claim 13 , wherein the consistency group objects are used in connection with storage operations including one or more of replication, rebuilding, or migration.
15. The method of claim 14 , wherein the consistency group objects are used in connection with restoring sets of disks across multiple virtual machines collectively requiring restoration of the same restored image after a disruption.
16. The method of claim 2 , wherein each of the consistency group objects is layered on top of a respective directory and is formed as a super-type of directory that incorporates functionality of underlying directory objects.
17. The method of claim 2 , wherein the consistency group objects are distinct objects each having structure and functionality similar to corresponding structure and functionality of the directories but not utilizing or relying on the directories for the structure and functionality.
18. The method of claim 2 , wherein the consistency group operations performed on the consistency group objects include CreateObject, DeleteObject, CreateReferenceLink, DeleteReferenceLink, CreateReplica, EnumerateObjects, and EnumerateSnapsets, wherein a reference link is a reference entry linking two shared objects such that an update to either object is visible on both objects.
19. The computer of claim 1 , wherein the filesystem objects of each consistency group are distinct filesystem objects independently storing respective distinct portions of the application data.