Granted Patent B1

US 10,936,538 · App. 16/883,922 · Granted Mar 2, 2021

Inventors: Thomas R. Unger (Seattle, WA); Michael Patrick Kirby (Lynnwood, WA); Thomas Scott Urban (Seattle, WA); Michael R. Fitz (Seattle, WA); Boyu Zhang (Seattle, WA)

Assignee: Qumulo, Inc.

G06F16/11G06F16/182

Abstract

Claims (64)

1. A method for managing data in a file system over a network using one or more processors that execute instructions to perform actions, comprising:

providing a query to determine one or more characteristics associated with a portion of a file system and a sampling model;

employing the sampling model to determine one or more objects in the file system based on a traversal of the portion of the file system, wherein a path of the traversal is based on the sampling model and the file system;

employing the sampling model to determine one or more metrics associated with the one or more alternate data streams (ADSs) included in the one or more objects, wherein each of the one or more objects that includes a separate main data stream and the one or more ADSs is associated with one or more data blocks and the one or more ADSs are each associated with one or more other data blocks;

employing the sampling model and the query to sample the one or more objects to provide one or more partial results;

distributing the sampling between the one or more objects based on the sampling model and the one or more metrics by performing further actions, including:

determining one or more first metrics of a first object, wherein the one or more first metrics are associated with each ADS that is included in the first object;

determining one or more second metrics of a second object, wherein the one or more second metrics are associated with each ADS that is included in the second object; and

determining a number of first samples to measure from the first object based on a ratio of the one or more first metrics and the one or more second metrics, wherein the number of first samples measured from the first object is greater than a number of second samples measured from the second object when values of the one or more first metrics are greater than values of the one or more second metrics; and

providing a result to the query based on the one or more partial results.

2. The method of claim 1 , wherein the sampling model includes one or more rules for a statistically fair sampling of the one or more objects based on the one or more metrics associated with the one or more ADSs.

3. The method of claim 1 , wherein the one or more metrics associated with the one or more alternate data streams (ADSs), further comprise: one or more of an amount of ADSs in the object, or, a number of data blocks included in each ADSs in the object.

4. The method of claim 1 , further comprising, terminating a descent traversal of the file system based on the one or more metrics matching one or more conditions defined in the sampling model, wherein the one or more conditions include, one or more values of the one or more metrics are less than a threshold value, an expiry of a timeout, a number of visited objects exceed a threshold value, or a proportion of visited objects relative to a number of objects in the portion of the file system exceed a threshold value.

5. A system for managing data in a file system comprising:

a network computer, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including:

providing a query to determine one or more characteristics associated with a portion of a file system and a sampling model;

employing the sampling model to determine one or more objects in the file system based on a traversal of the portion of the file system, wherein a path of the traversal is based on the sampling model and the file system;

employing the sampling model to determine one or more metrics associated with the one or more alternate data streams (ADSs) included in the one or more objects, wherein each of the one or more objects that includes a separate main data stream and the one or more ADSs is associated with one or more data blocks and the one or more ADSs are each associated with one or more other data blocks;

employing the sampling model and the query to sample the one or more objects to provide one or more partial results, wherein the sampling is distributed between the one or more objects based on the sampling model and the one or more metrics;

employing the sampling model and the query to sample the one or more objects to provide one or more partial results;

distributing the sampling between the one or more objects based on the sampling model and the one or more metrics by performing further actions, including:

determining one or more first metrics of a first object, wherein the one or more first metrics are associated with each ADS that is included in the first object;

determining one or more second metrics of a second object, wherein the one or more second metrics are associated with each ADS that is included in the second object; and

determining a number of first samples to measure from the first object based on a ratio of the one or more first metrics and the one or more second metrics, wherein the number of first samples measured from the first object is greater than a number of second samples measured from the second object when values of the one or more first metrics are greater than values of the one or more second metrics; and

providing a result to the query based on the one or more partial results; and

a client computer, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including, providing the query.

6. The system of claim 5 , wherein the sampling model includes one or more rules for a statistically fair sampling of the one or more objects based on the one or more metrics associated with the one or more ADSs.

7. The system of claim 5 , wherein the one or more metrics associated with the one or more alternate data streams (ADSs), further comprise: one or more of an amount of ADSs in the object, or, a number of data blocks included in each ADSs in the object.

8. The system of claim 5 , wherein the one or more network computer processors execute instructions that perform actions, further comprising, terminating a descent traversal of the file system based on the one or more metrics matching one or more conditions defined in the sampling model, wherein the one or more conditions include, one or more values of the one or more metrics are less than a threshold value, an expiry of a timeout, a number of visited objects exceed a threshold value, or a proportion of visited objects relative to a number of objects in the portion of the file system exceed a threshold value.

9. A processor readable non-transitory storage media that includes instructions for managing data in a file system over a network, wherein execution of the instructions by one or more processors on one or more network computers performs actions, comprising:

providing a query to determine one or more characteristics associated with a portion of a file system and a sampling model;

employing the sampling model to determine one or more objects in the file system based on a traversal of the portion of the file system, wherein a path of the traversal is based on the sampling model and the file system;

employing the sampling model to determine one or more metrics associated with the one or more alternate data streams (ADSs) included in the one or more objects, wherein each of the one or more objects that includes a separate main data stream and the one or more ADSs is associated with one or more data blocks and the one or more ADSs are each associated with one or more other data blocks;

employing the sampling model and the query to sample the one or more objects to provide one or more partial results;

distributing the sampling between the one or more objects based on the sampling model and the one or more metrics by performing further actions, including:

determining one or more first metrics of a first object, wherein the one or more first metrics are associated with each ADS that is included in the first object;

determining one or more second metrics of a second object, wherein the one or more second metrics are associated with each ADS that is included in the second object; and

determining a number of first samples to measure from the first object based on a ratio of the one or more first metrics and the one or more second metrics, wherein the number of first samples measured from the first object is greater than a number of second samples measured from the second object when values of the one or more first metrics are greater than values of the one or more second metrics; and

providing a result to the query based on the one or more partial results.

10. The media of claim 9 , wherein the sampling model includes one or more rules for a statistically fair sampling of the one or more objects based on the one or more metrics associated with the one or more ADSs.

11. The media of claim 9 , wherein the one or more metrics associated with the one or more alternate data streams (ADSs), further comprise: one or more of an amount of ADSs in the object, or, a number of data blocks included in each ADSs in the object.

12. The media of claim 9 , further comprising, terminating a descent traversal of the file system based on the one or more metrics matching one or more conditions defined in the sampling model, wherein the one or more conditions include, one or more values of the one or more metrics are less than a threshold value, an expiry of a timeout, a number of visited objects exceed a threshold value, or a proportion of visited objects relative to a number of objects in the portion of the file system exceed a threshold value.

13. A network computer for managing data in a file system, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including:

providing a query to determine one or more characteristics associated with a portion of a file system and a sampling model;

employing the sampling model to determine one or more objects in the file system based on a traversal of the portion of the file system, wherein a path of the traversal is based on the sampling model and the file system;

employing the sampling model to determine one or more metrics associated with the one or more alternate data streams (ADSs) included in the one or more objects, wherein each of the one or more objects that includes a separate main data stream and the one or more ADSs is associated with one or more data blocks and the one or more ADSs are each associated with one or more other data blocks;

employing the sampling model and the query to sample the one or more objects to provide one or more partial results;

distributing the sampling between the one or more objects based on the sampling model and the one or more metrics by performing further actions, including:

determining one or more first metrics of a first object, wherein the one or more first metrics are associated with each ADS that is included in the first object determining one or more second metrics of a second object, wherein

the one or more second metrics are associated with each ADS that is included in the second object; and

determining a number of first samples to measure from the first object based on a ratio of the one or more first metrics and the one or more second metrics, wherein the number of first samples measured from the first object is greater than a number of second samples measured from the second object when values of the one or more first metrics are greater than values of the one or more second metrics; and

providing a result to the query based on the one or more partial results.

14. The network computer of claim 13 , wherein the sampling model includes one or more rules for a statistically fair sampling of the one or more objects based on the one or more metrics associated with the one or more ADSs.

15. The network computer of claim 13 , wherein the one or more metrics associated with the one or more alternate data streams (ADSs), further comprise: one or more of an amount of ADSs in the object, or, a number of data blocks included in each ADSs in the object.

16. The network computer of claim 13 , wherein the one or more processors execute instructions that perform actions, further comprising, terminating a descent traversal of the file system based on the one or more metrics matching one or more conditions defined in the sampling model, wherein the one or more conditions include, one or more values of the one or more metrics are less than a threshold value, an expiry of a timeout, a number of visited objects exceed a threshold value, or a proportion of visited objects relative to a number of objects in the portion of the file system exceed a threshold value.

Assignments (2)

Continuity (1)

Cited By (9)

US 12,222,903 US 12,292,853 US 12,346,290 US 12,443,559 US 12,443,568 US 12,481,625 US 12,585,563 US 12,619,582 US 12,670,081