Granted Patent B1

US 11,934,660 · App. 18/503,918 · Granted Mar 19, 2024

Inventors: Matthew Christopher McMullan (Bellevue, WA); Aaron James Passey (San Rafael, CA); Jonathan Michael MacLaren (Seattle, WA); Yuxi Bai (Seattle, WA); Thomas Gregory Rothschilds (Seattle, WA); Michael Anthony Chmiel (Seattle, WA); Tyler Morrison Moody (Seattle, WA); Pathirat Kosakanchit (Seattle, WA); Rowan Arthur Phipps (Seattle, WA)

Assignee: Qumulo, Inc.

G06F3/0613G06F3/0607G06F3/064G06F3/0644G06F3/0683

Abstract

Claims (111)

1. A method for managing data in a file system over a network using one or more processors to execute instructions that are configured to cause performance of actions, comprising:

determining a write tier in the file system for storing data associated with one or more write requests from one or more file system clients based on location information associated with the one or more write requests, wherein the file system includes the write tier and an object tier;

predicting a value for a performance metric that corresponds to the one or more write requests based on one or more characteristics of the one or more write requests, wherein the performance metric is determined based on a plurality of interactions with the write tier; and

employing the predicted value that exceeds a threshold value of the performance metric to cause performance of further actions, including:

queuing a portion of the one or more write requests in a memory buffer that is separate from the write tier based on the predicted value and the threshold value;

combining the queued portion of the one or more write requests into one or more data segments; and

storing the one or more data segments in the write tier, wherein a measured value of the performance metric is less than the threshold value, and wherein the measured valued is associated with storing the one or more data segments in the write tier.

2. The method of claim 1 , wherein combining the portion of the one or more write requests further comprises:

generating one or more Write Ahead Log (WAL) meta-data blocks and one or more payload blocks based on the queued portion of the one or more write requests, wherein the data associated with the queued portion of the write requests is included in the one or more payload blocks, and wherein meta-data associated with the one or more payload blocks is included in the one or more WAL meta-data blocks; and

including the one or more WAL meta-data blocks and the one or more payload blocks in the one or more data segments.

3. The method of claim 1 , wherein storing the one or more data segments, further comprises:

storing one or more payload blocks included in the one or more data segments in a payload segment of the write tier; and

storing one or more Write Ahead Log (WAL) meta-data blocks included in the one or more data segments in a WAL segment of the write tier.

4. The method of claim 1 , wherein predicting the value for the performance metric, further comprises:

predicting an input-output-operations-per-second (IOPS) value associated with interactions with the write tier based on one or more characteristics of the one or more write requests.

5. The method of claim 1 , wherein determining the write tier in the file system for storing data associated with one or more write requests, further comprises:

determining one or more portions of data associated with the one or more write requests to store in the object tier, wherein storing the one or more portions of data in the object tier bypasses the write tier.

6. The method of claim 1 , further comprising:

monitoring an amount of available storage space in the write tier; and

in response to the amount of available storage space being less than threshold value, causing performance of other further actions, including:

combining one or more WAL meta-data blocks and one or more payload blocks stored in the one or more data segments into one or more objects; and

storing the one or more objects in the object tier to increase the available space in the write tier.

7. The method of claim 1 , further comprising:

employing a cloud-based block storage service to provide capacity to store data for the write tier.

8. The method of claim 1 , further comprising:

determining a read tier in the file system that is storing a portion of the data associated with the one or more write requests; and

invalidating or updating the portion of the data in the read tier.

9. A system for managing data in a file system comprising:

a network computer, comprising:

a memory that stores at least instructions; and

one or more processors that execute instructions that are configured to cause performance of actions, including:

determining a write tier in the file system for storing data associated with one or more write requests from one or more file system clients based on location information associated with the one or more write requests, wherein the file system includes the write tier and an object tier;

predicting a value for a performance metric that corresponds to the one or more write requests based on one or more characteristics of the one or more write requests, wherein the performance metric is determined based on a plurality of interactions with the write tier; and

employing the predicted value that exceeds a threshold value of the performance metric to cause performance of further actions, including:

queuing a portion of the one or more write requests in a memory buffer that is separate from the write tier based on the predicted value and the threshold value;

combining the queued portion of the one or more write requests into one or more data segments; and

storing the one or more data segments in the write tier, wherein a measured value of the performance metric is less than the threshold value, and wherein the measured valued is associated with storing the one or more data segments in the write tier; and

one or more client computers, comprising:

a memory that stores at least instructions; and

one or more processors that execute instructions that are configured to cause performance of other actions, including:

providing the one or more write requests to the file system.

10. The system of claim 9 , wherein combining the portion of the one or more write requests further comprises:

generating one or more Write Ahead Log (WAL) meta-data blocks and one or more payload blocks based on the queued portion of the one or more write requests, wherein the data associated with the queued portion of the write requests is included in the one or more payload blocks, and wherein meta-data associated with the one or more payload blocks is included in the one or more WAL meta-data blocks; and

including the one or more WAL meta-data blocks and the one or more transaction blocks in the one or more data segments.

11. The system of claim 9 , wherein storing the one or more data segments, further comprises:

storing one or more payload blocks included in the one or more data segments in a payload segment of the write tier; and

storing one or more Write Ahead Log (WAL) meta-data blocks included in the one or more data segments in a WAL segment of the write tier.

12. The system of claim 9 , wherein predicting the value for the performance metric, further comprises:

predicting an input-output-operations-per-second (IOPS) value associated with interactions with the write tier based on one or more characteristics of the one or more write requests.

13. The system of claim 9 , wherein determining the write tier in the file system for storing data associated with one or more write requests, further comprises:

determining one or more portions of data associated with the one or more write requests to store in the object tier, wherein storing the one or more portions of data in the object tier bypasses the write tier.

14. The system of claim 9 , wherein the one or more network computer processors execute instructions that are configured to cause performance of actions, further comprising:

monitoring an amount of available storage space in the write tier; and

in response to the amount of available storage space being less than threshold value, causing performance of other further actions, including:

combining one or more other payload segments stored in the write tier into one or more objects; and

storing the one or more objects in an object tier, wherein the amount of available storage space in the write tier is increased based on a size of the one or more objects.

15. The system of claim 9 , wherein the one or more network computer processors execute instructions that are configured to cause performance of actions, further comprising:

employing a cloud-based block storage service to provide capacity to store data for the write tier.

16. The system of claim 9 , wherein the one or more network computer processors execute instructions that are configured to cause performance of actions, further comprising:

determining a read tier in the file system that is storing a portion of the data associated with the one or more write requests; and

invalidating the portion of the data in the read tier.

17. A processor readable non-transitory storage media that includes instructions for managing data in a file system over a network, wherein execution of the instructions by one or more processors on one or more network computers performs actions, comprising:

determining a write tier in the file system for storing data associated with one or more write requests from one or more file system clients based on location information associated with the one or more write requests, wherein the file system includes the write tier and an object tier;

predicting a value for a performance metric that corresponds to the one or more write requests based on one or more characteristics of the one or more write requests, wherein the performance metric is determined based on a plurality of interactions with the write tier; and

employing the predicted value that exceeds a threshold value of the performance metric to cause performance of further actions, including:

queuing a portion of the one or more write requests in a memory buffer that is separate from the write tier based on the predicted value and the threshold value;

combining the queued portion of the one or more write requests into one or more data segments; and

storing the one or more data segments in the write tier, wherein a measured value of the performance metric is less than the threshold value, and wherein the measured valued is associated with storing the one or more data segments in the write tier.

18. The media of claim 17 , wherein combining the portion of the one or more write requests further comprises:

generating one or more Write Ahead Log (WAL) meta-data blocks and one or more payload blocks based on the queued portion of the one or more write requests, wherein the data associated with the queued portion of the write requests is included in the one or more payload blocks, and wherein meta-data associated with the one or more payload blocks is included in the one or more WAL payload blocks; and

including the one or more WAL meta-data blocks and the one or more payload blocks in the one or more data segments.

19. The media of claim 17 , wherein storing the one or more data segments, further comprises:

storing one or more transaction blocks included in the one or more data segments in a payload segment of the write tier; and

storing one or more Write Ahead Log (WAL) blocks included in the one or more data segments in a WAL segment of the write tier.

20. The media of claim 17 , wherein predicting the value for the performance metric, further comprises:

predicting an input-output-operations-per-second (IOPS) value associated with interactions with the write tier based on one or more characteristics of the one or more write requests.

21. The media of claim 17 , wherein determining the write tier in the file system for storing data associated with one or more write requests, further comprises:

determining one or more portions of data associated with the one or more write requests to store in the object tier, wherein storing the one or more portions of data in the object tier bypasses the write tier.

22. The media of claim 17 , further comprising:

monitoring an amount of available storage space in the write tier; and

in response to the amount of available storage space being less than threshold value, causing performance of other further actions, including:

combining one or more other payload segments stored in the write tier into one or more objects; and

storing the one or more objects in an object tier, wherein the amount of available storage space in the write tier is increased based on a size of the one or more objects.

23. The media of claim 17 , further comprising:

employing a cloud-based block storage service to provide capacity to store data for the write tier.

24. A network computer for managing data in a file system, comprising:

a memory that stores at least instructions; and

one or more processors that execute instructions that are configured to cause performance of actions, including:

determining a write tier in the file system for storing data associated with one or more write requests from one or more file system clients based on location information associated with the one or more write requests, wherein the file system includes the write tier and the object tier;

predicting a value for a performance metric that corresponds to the one or more write requests based on one or more characteristics of the one or more write requests, wherein the performance metric is determined based on a plurality of interactions with the write tier; and

employing the predicted value that exceeds a threshold value of the performance metric to cause performance of further actions, including:

queuing a portion of the one or more write requests in a memory buffer that is separate from the write tier based on the predicted value and the threshold value;

combining the queued portion of the one or more write requests into one or more data segments; and

storing the one or more data segments in the write tier, wherein a measured value of the performance metric is less than the threshold value, and wherein the measured valued is associated with storing the one or more data segments in the write tier.

25. The network computer of claim 24 , wherein combining the portion of the one or more write requests further comprises:

generating one or more Write Ahead Log (WAL) meta-data blocks and one or more payload blocks based on the queued portion of the one or more write requests, wherein the data associated with the queued portion of the write requests is included in the one or more payload blocks, and wherein meta-data associated with the one or more payload blocks is included in the one or more WAL meta-data blocks; and

including the one or more WAL meta-data blocks and the one or more payload blocks in the one or more data segments.

26. The network computer of claim 24 , wherein storing the one or more data segments, further comprises:

storing one or more payload blocks included in the one or more data segments in a payload segment of the write tier; and

storing one or more Write Ahead Log (WAL) blocks included in the one or more data segments in a WAL segment of the write tier.

27. The network computer of claim 24 , wherein predicting the value for the performance metric, further comprises:

predicting an input-output-operations-per-second (IOPS) value associated with interactions with the write tier based on one or more characteristics of the one or more write requests.

28. The network computer of claim 24 , wherein determining the write tier in the file system for storing data associated with one or more write requests, further comprises:

determining one or more portions of data associated with the one or more write requests to store in the object tier, wherein storing the one or more portions of data in the object tier bypasses the write tier.

29. The network computer of claim 24 , wherein the one or more processors execute instructions that are configured to cause performance of actions, further comprising:

monitoring an amount of available storage space in the write tier; and

in response to the amount of available storage space being less than threshold value, causing performance of other further actions, including:

combining one or more other payload segments stored in the write tier into one or more objects; and

storing the one or more objects in an object tier, wherein the amount of available storage space in the write tier is increased based on a size of the one or more objects.

30. The network computer of claim 24 , wherein the one or more processors execute instructions that are configured to cause performance of actions, further comprising:

employing a cloud-based block storage service to provide capacity to store data for the write tier.

Assignments (2)

Cited By (12)

US 12,222,903 US 12,292,853 US 12,346,290 US 12,443,559 US 12,443,568 US 12,481,625 US 12,585,563 US 12,619,582 US 12,625,846 US 12,670,081 US 12,743,357 US 12,748,730