Granted Patent B1

US 11,599,508 · App. 17/588,895 · Granted Mar 7, 2023

Inventors: Christopher Charles Harward (Vancouver, CA); Kevin David Jamieson (North Vancouver, CA); Aleksei Martynov (Seattle, WA); Adam Mitha (North Vancouver, CA); Pavel Shilovskiy (Sammamish, WA); Sihang Su (Vancouver, CA); Colin Wallace (Seattle, WA)

Assignee: Qumulo, Inc.

G06F16/185G06F16/182

Abstract

Claims (116)

1. A method for managing data in a file system over a network using one or more processors that execute instructions to perform actions, comprising:

providing a file system that includes a plurality of directories;

determining a source bucket on a cloud computing environment, wherein the source bucket is an object store that contains a plurality of objects;

associating the source bucket with a target directory in the file system based on an integration relationship, wherein the integration relationship includes one or more of an identity of the source bucket, or the path to the target directory;

determining one or more filter rules to access the source bucket or the target directory based on the identity of the source bucket or the path to the target directory included in the integration relationship;

providing one or more object keys from the object store based on a query and the one or more filter rules, wherein each object key corresponds to an object in the source bucket that conforms to the one or more filter rules;

determining one or more source objects in the source bucket based on the one or more object keys, wherein each source object corresponds to one of the one or more object keys;

determining one or more target files that correspond to the one or more source objects based on the integration relationship; and

in response to determining one or more changed target files or one or more changed source objects, copying the one or more source objects that correspond to the one or more changed source objects or the one or more changed target files to the target directory.

2. The method of claim 1 , further comprising:

determining one or more object profiles based on the one or more objects, wherein each object profile corresponds to a source object in the source bucket and a target file in the target directory, and wherein each object profile includes one or more characteristics associated with the source object and the target file.

3. The method of claim 1 , wherein copying the one or more source objects to the target directory, further comprises:

generating an import job based on the integration relationship;

determining one or more portions of the one or more object keys, wherein the one or more portions are provided by the cloud computing environment based on a query;

generating one or more worker jobs that are enabled to copy the one or more source objects to the target directory;

associating each worker job to a portion of the one or more object keys; and

executing each worker job to copy a portion of the one or more source objects to the target directory.

4. The method of claim 1 , wherein determining the one or more object keys from the object store based on the one or more filter rules, further comprises:

providing the one or more filter rules to the cloud computing environment, wherein the one or more filter rules includes a filter rule to match a defined string pattern;

determining one or more matched source objects that are associated with a label that is matched by the filter rule; and

determining the one or more object keys based on a correspondence of the one or more object keys and the one or more matched source objects.

5. The method of claim 1 , further comprising:

in response to copying each of the one or more source objects that correspond to one or more of the one or more changed source objects or the one or more changed target files to the target directory, executing the query with the one or more filters to determine one or more additional object keys; and

in response to determining the one or more additional object keys, performing further additional actions, including:

determining one or more additional source objects in the source bucket based on the one or more additional object keys;

determining one or more additional target files that correspond to the one or more additional source objects based on the integration relationship; and

in response to determining one or more changed additional target files or one or more changed additional source objects, copying the one or more additional source objects that correspond to the one or more additional changed source objects or the one or more additional changed target files to the target directory.

6. A network computer for managing data in a file system, comprising:

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including:

providing a file system that includes a plurality of directories;

determining a source bucket on a cloud computing environment, wherein the source bucket is an object store that contains a plurality of objects;

associating the source bucket with a target directory in the file system based on an integration relationship, wherein the integration relationship includes one or more of an identity of the source bucket, or the path to the target directory;

determining one or more filter rules to access the source bucket or the target directory based on the identity of the source bucket or the path to the target directory included in the integration relationship;

providing one or more object keys from the object store based on a query and the one or more filter rules, wherein each object key corresponds to an object in the source bucket that conforms to the one or more filter rules;

determining one or more source objects in the source bucket based on the one or more object keys, wherein each source object corresponds to one of the one or more object keys;

determining one or more target files that correspond to the one or more source objects based on the integration relationship; and

in response to determining one or more changed target files or one or more changed source objects, copying the one or more source objects that correspond to the one or more changed source objects or the one or more changed target files to the target directory.

7. The network computer of claim 6 , wherein the one or more processors execute instructions that perform actions, further comprising:

determining one or more object profiles based on the one or more objects, wherein each object profile corresponds to a source object in the source bucket and a target file in the target directory, and wherein each object profile includes one or more characteristics associated with the source object and the target file.

8. The network computer of claim 6 , wherein copying the one or more source objects to the target directory, further comprises:

generating an import job based on the integration relationship;

determining one or more portions of the one or more object keys, wherein the one or more portions are provided by the cloud computing environment based on a query;

generating one or more worker jobs that are enabled to copy the one or more source objects to the target directory;

associating each worker job to a portion of the one or more object keys; and

executing each worker job to copy a portion of the one or more source objects to the target directory.

9. The network computer of claim 6 , wherein determining the one or more object keys from the object store based on the one or more filter rules, further comprises:

providing the one or more filter rules to the cloud computing environment, wherein the one or more filter rules includes a filter rule to match a defined string pattern;

determining one or more matched source objects that are associated with a label that is matched by the filter rule; and

determining the one or more object keys based on a correspondence of the one or more object keys and the one or more matched source objects.

10. The network computer of claim 6 , wherein the one or more processors execute instructions that perform actions, further comprising:

in response to copying each of the one or more source objects that correspond to one or more of the one or more changed source objects or the one or more changed target files to the target directory, executing the query with the one or more filters to determine one or more additional object keys; and

in response to determining the one or more additional object keys, performing further additional actions, including:

determining one or more additional source objects in the source bucket based on the one or more additional object keys;

determining one or more additional target files that correspond to the one or more additional source objects based on the integration relationship; and

in response to determining one or more changed additional target files or one or more changed additional source objects, copying the one or more additional source objects that correspond to the one or more additional changed source objects or the one or more additional changed target files to the target directory.

11. A processor readable non-transitory storage media that includes instructions for managing data in a file system over a network, wherein execution of the instructions by one or more processors on one or more network computers performs actions, comprising:

providing a file system that includes a plurality of directories;

determining a source bucket on a cloud computing environment, wherein the source bucket is an object store that contains a plurality of objects;

associating the source bucket with a target directory in the file system based on an integration relationship, wherein the integration relationship includes one or more of an identity of the source bucket, or the path to the target directory;

determining one or more filter rules to access the source bucket or the target directory based on the identity of the source bucket or the path to the target directory included in the integration relationship;

providing one or more object keys from the object store based on a query and the one or more filter rules, wherein each object key corresponds to an object in the source bucket that conforms to the one or more filter rules;

determining one or more source objects in the source bucket based on the one or more object keys, wherein each source object corresponds to one of the one or more object keys;

determining one or more target files that correspond to the one or more source objects based on the integration relationship; and

in response to determining one or more changed target files or one or more changed source objects, copying the one or more source objects that correspond to the one or more changed source objects or the one or more changed target files to the target directory.

12. The media of claim 11 , further comprising:

determining one or more object profiles based on the one or more objects, wherein each object profile corresponds to a source object in the source bucket and a target file in the target directory, and wherein each object profile includes one or more characteristics associated with the source object and the target file.

13. The media of claim 11 , wherein copying the one or more source objects to the target directory, further comprises:

generating an import job based on the integration relationship;

determining one or more portions of the one or more object keys, wherein the one or more portions are provided by the cloud computing environment based on a query;

generating one or more worker jobs that are enabled to copy the one or more source objects to the target directory;

associating each worker job to a portion of the one or more object keys; and

executing each worker job to copy a portion of the one or more source objects to the target directory.

14. The media of claim 11 , wherein determining the one or more object keys from the object store based on the one or more filter rules, further comprises:

providing the one or more filter rules to the cloud computing environment, wherein the one or more filter rules includes a filter rule to match a defined string pattern;

determining one or more matched source objects that are associated with a label that is matched by the filter rule; and

determining the one or more object keys based on a correspondence of the one or more object keys and the one or more matched source objects.

15. The media of claim 11 , further comprising:

in response to copying each of the one or more source objects that correspond to one or more of the one or more changed source objects or the one or more changed target files to the target directory, executing the query with the one or more filters to determine one or more additional object keys; and

in response to determining the one or more additional object keys, performing further additional actions, including:

determining one or more additional source objects in the source bucket based on the one or more additional object keys;

determining one or more additional target files that correspond to the one or more additional source objects based on the integration relationship; and

in response to determining one or more changed additional target files or one or more changed additional source objects, copying the one or more additional source objects that correspond to the one or more additional changed source objects or the one or more additional changed target files to the target directory.

16. A system for managing data in a file system comprising:

a network computer, comprising:

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including:

providing a file system that includes a plurality of directories;

determining a source bucket on a cloud computing environment, wherein the source bucket is an object store that contains a plurality of objects;

associating the source bucket with a target directory in the file system based on an integration relationship, wherein the integration relationship includes one or more of an identity of the source bucket, or the path to the target directory;

determining one or more filter rules to access the source bucket or the target directory based on the identity of the source bucket or the path to the target directory included in the integration relationship;

providing one or more object keys from the object store based on a query and the one or more filter rules, wherein each object key corresponds to an object in the source bucket that conforms to the one or more filter rules;

determining one or more source objects in the source bucket based on the one or more object keys, wherein each source object corresponds to one of the one or more object keys;

determining one or more target files that correspond to the one or more source objects based on the integration relationship; and

in response to determining one or more changed target files or one or more changed source objects, copying the one or more source objects that correspond to the one or more changed source objects or the one or more changed target files to the target directory; and

a client computer, comprising:

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including, providing one or more of the one or more integration relationships or the one or more filter rules.

17. The system of claim 16 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:

determining one or more object profiles based on the one or more objects, wherein each object profile corresponds to a source object in the source bucket and a target file in the target directory, and wherein each object profile includes one or more characteristics associated with the source object and the target file.

18. The system of claim 16 , wherein copying the one or more source objects to the target directory, further comprises:

generating an import job based on the integration relationship;

determining one or more portions of the one or more object keys, wherein the one or more portions are provided by the cloud computing environment based on a query;

generating one or more worker jobs that are enabled to copy the one or more source objects to the target directory;

associating each worker job to a portion of the one or more object keys; and

executing each worker job to copy a portion of the one or more source objects to the target directory.

19. The system of claim 16 , wherein determining the one or more object keys from the object store based on the one or more filter rules, further comprises:

providing the one or more filter rules to the cloud computing environment, wherein the one or more filter rules includes a filter rule to match a defined string pattern;

determining one or more matched source objects that are associated with a label that is matched by the filter rule; and

determining the one or more object keys based on a correspondence of the one or more object keys and the one or more matched source objects.

20. The system of claim 16 , wherein the one or more network computer processors execute instructions that perform actions, further comprising:

in response to copying each of the one or more source objects that correspond to one or more of the one or more changed source objects or the one or more changed target files to the target directory, executing the query with the one or more filters to determine one or more additional object keys; and

in response to determining the one or more additional object keys, performing further additional actions, including:

determining one or more additional source objects in the source bucket based on the one or more additional object keys;

determining one or more additional target files that correspond to the one or more additional source objects based on the integration relationship; and

in response to determining one or more changed additional target files or one or more changed additional source objects, copying the one or more additional source objects that correspond to the one or more additional changed source objects or the one or more additional changed target files to the target directory.

Assignments (3)

Cited By (10)

US 12,222,903 US 12,292,853 US 12,346,290 US 12,443,559 US 12,443,568 US 12,481,625 US 12,585,563 US 12,619,582 US 12,625,846 US 12,670,081