Partitioning a distributed filesystem namespace
The described technology is generally directed towards domains that data services can use to collect files of a global filesystem namespace into groups. A data service (a domain patron) creates a domain, e.g., for a particular directory, and a domain manager associates files under that directory with domain membership information. Thereafter, the data service can use the domain membership information associated with a file to determine whether to include that file in a data service operation. In one implementation the membership information is maintained in file metadata, facilitating fast and efficient retrieval of the information in near constant time. Exclusion from a domain, hard links, renames and alternate data streams are among the various aspects supported by the domain technology.
1. A method, comprising:
receiving, by a system comprising a processor, a first request to create a domain on a directory of a filesystem;
in response to the first request,
creating the domain on the directory,
associating the directory with the domain, and
in response to the directory not being empty, associating any nodes under the directory with the domain;
receiving a second request to read from a binary large object of data stored in the domain on the directory;
in response to the second request, performing an iterative read of the binary large object of data, wherein a lock is held for a duration of the iterative read to prevent the binary large object of data from changing during the iterative read;
receiving a third request with respect to a file object stored in one of the nodes under the directory; and
in response to the third request,
determining whether domain membership information associated with the file object is up to date,
in response to the domain membership information of the file object being determined to be up to date, returning the domain membership information, and
in response to the domain membership information of the file object being determined to be out of date, updating the domain membership information by obtaining updated domain membership information from a parent node of the file object, and returning the updated domain membership, wherein the domain membership information is cached in a domain cache, and wherein invalidation of the domain cache is performed using a global lock of each domain being invalidated.
2. The method of claim 1 , further comprising, in further response to the first request, storing an identifier of the domain in a domain data structure.
3. The method of claim 1 , further comprising, in further response to the first request, updating a generation value in a domain data structure.
4. The method of claim 1 , wherein the directory is not empty, and wherein the associating the any of the nodes under the directory with the domain comprises tagging respective any attributes associated with the any of the nodes with an identifier of the domain to designate that the any of the nodes are members of the domain.
5. The method of claim 1 , wherein the directory is not empty, wherein the node comprises the file object, and wherein the associating the any of the nodes under the directory with the domain comprises traversing a directory hierarchy associated with the directory, and tagging a file attribute associated with the file object with an identifier of the domain to designate that the any of the nodes are members of the domain when the file object is reached during the traversing.
6. The method of claim 1 , wherein the directory is an empty directory, and further comprising, detecting a new node added to the directory, and in response to the detecting, associating the new node with the domain.
7. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
in response to a first request to create a domain on a directory of a filesystem,
creating the domain on the directory,
associating the directory with the domain, and
in response to the directory not being empty, associating any nodes under the directory with the domain;
receiving a second request to read from a binary large object of data stored in the domain on the directory;
in response to the second request, performing an iterative read of the binary large object of data, wherein a lock is held for a duration of the iterative read to prevent the binary large object of data from changing during the iterative read;
receiving a third request with respect to a file object stored in one of the nodes under the directory; and
in response to the third request, determining whether domain membership information associated with the file object is up to date,
in response to the domain membership information of the file object being determined to be up to date, returning the domain membership information, and
in response to the domain membership information of the file object being determined to be out of date, updating the domain membership information, the updating comprising obtaining updated domain membership information associated with a parent node of the file object, and returning the updated domain membership, wherein the domain membership information is cached in a domain cache, and wherein invalidation of the domain cache is performed using a global lock of each domain being invalidated.
8. The system of claim 7 , wherein the associating the directory with the domain comprises associating the directory with the domain membership information, and associating the file object with the domain membership information.
9. The system of claim 7 , wherein the operations further comprise, maintaining the domain membership information in association with the file object comprising maintaining a domain identifier of the domain in metadata of the file object.
10. The system of claim 7 , wherein the operations further comprise, detecting an operation that renames the directory above the file object to a namespace corresponding to a second domain, obtaining first domain membership data of the first domain and second domain membership data of the second domain, and, in response to the first domain membership data being determined to be different from the second membership data, updating the domain membership information of the file object based on the second membership data.
11. The system of claim 7 , wherein the directory is a first directory, wherein the domain is a first domain, and wherein the operations further comprise, detecting an operation that provides a hard link from the first directory to a second directory corresponding to a second domain, and updating the domain membership information associated with the file object based on domain data associated with the second domain.
12. The system of claim 11 , wherein the operations further comprise, detecting an operation that removes the hard link between the first directory and the second directory, and updating the domain membership information associated with the file object based on the file object being determined to be no longer hard linked via the first directory to the second directory.
13. The system of claim 7 , wherein the domain membership information of the file object comprises a first domain identifier and a second domain identifier, and wherein the operations further comprise, adding a third domain identifier to the domain membership information, in which the third domain identifier logically excludes the first domain identifier from the domain membership information.
14. The system of claim 13 , wherein the returning the domain membership information comprises generating a response by excluding the first identifier from the response based on the third identifier, and returning the response comprising the second identifier.
15. The system of claim 7 , wherein the domain membership information of the file object comprises a first domain identifier, and wherein the operations further comprise, maintaining the first domain identifier in a domain data structure in conjunction with data service-specific data.
16. The system of claim 7 , wherein the determining whether the domain membership information of the file object is up to date comprises evaluating a domain membership information generation number associated with the domain membership information of the file object against a generation counter value corresponding to a most-recent domain creation.
17. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, the operations comprising:
in response to a first request from a data service,
creating a domain in association with a directory of a filesystem,
associating the directory with a domain identifier,
in response to the directory not being empty, associating any nodes under the directory with the domain identifier,
receiving a second request to read from a binary large object of data stored in the domain on the directory;
in response to the second request, performing an iterative read of the binary large object of data, wherein a lock is held until the iterative read is complete to prevent changes being made to the binary large object of data during the iterative read;
receiving a third request with respect to a file object stored in one of the nodes under the directory; and
in response to the third request,
determining whether domain membership information associated with the file object is up to date,
in response to the domain membership information of the file object being determined to be up to date, returning the domain membership information, and
in response to the domain membership information of the file object being determined to be out of date, updating the domain membership information by obtaining updated domain membership information from a parent node of the file object, and returning the updated domain membership, wherein the domain membership information is cached in a domain cache, and wherein invalidation of the domain cache is performed using a global lock of each domain being invalidated.
18. The non-transitory machine-readable medium of claim 17 , wherein the associating the any of the nodes under the directory with the domain identifier comprises tagging the any of the nodes with the domain identifier, and wherein the operations further comprise, determining that the domain membership information is up to date by determining that no tagging operation is outstanding with respect to the directory and that no renaming operation is outstanding with respect to the directory.
19. The non-transitory machine-readable medium of claim 17 , wherein the directory comprises a first directory, wherein the first directory has a hard link to a second directory, and wherein the determining the domain membership information of the file object comprises combining domain membership information based on the first directory and the second directory.
20. The non-transitory machine-readable medium of claim 17 , wherein the operations further comprise, in further response to the first request, storing the domain identifier in a domain data structure, and updating a generation value in the domain data structure.