Granted Patent B1

US 9,792,295 · App. 14/484,639 · Granted Oct 17, 2017

Inventor: Silvius V. Rus (Orinda, CA)

Assignee: Quantcast Corporation

G06F17/30194G06F3/067G06F17/30584

Abstract

Claims (60)

1. A computer-implemented method for distributing data stored in a distributed storage comprising a plurality of disks, the method comprising:

maintaining a plurality of logically independent file systems, wherein each file system includes a data set stored by the distributed storage;

determining file system access pattern levels by observing accesses to data files in the data set included in each of the file systems;

determining which file system is most frequently accessed among the plurality of logically independent file systems;

establishing a dedicated physical partition in each of the plurality of disks for the data set of the most frequently accessed file system, the dedicated physical partition located near an outer edge of the respective disk;

and distributing the data set of the most frequently accessed file system across dedicated physical partitions of the plurality of disks before distributing data sets of other file systems of the plurality of logically independent file systems across another partition in each of the plurality of disks, wherein the data sets of the other file systems are distributed based at least in part on determined file system access pattern levels for the plurality of logically independent file systems;

determining that a specific file system from a plurality of other logically independent file systems has an access pattern level specifying a higher probability of future access than a probability of future access specified by an access pattern level of another one of the plurality of logically independent file systems; and

redistributing a data set included in the specific file system prior to redistributing a data set included in another file system based on the determination that the specific file system from the plurality of other logically independent file systems has the access pattern level specifying a higher probability of future access than the probability of future access specified by the access pattern level of the another one of the plurality of logically independent file systems.

2. The computer-implemented method of claim 1 , wherein

distributing the data sets of the other logically independent file systems comprises:

redistributing a particular data set included in a particular file system from the plurality of logically independent file systems uniformly across a plurality of storage devices of the distributed storage.

3. The computer-implemented method of claim 1 , wherein distributing the data sets of the other logically independent file systems comprises:

determining storage requirements for the plurality of file systems;

obtaining device characteristic information for a plurality of storage devices of the distributed storage; and

redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage.

4. The computer-implemented method of claim 3 , wherein redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage comprises:

redistributing a specific data set included in a specific file system from the plurality of logically independent file systems across the plurality of storage devices in proportion to a performance characteristic for the specific file system described by the performance characteristic information.

5. The computer-implemented method of claim 3 , wherein redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage comprises:

calculating a target amount of storage space assigned to a specific file system for each of the plurality of storage devices by dividing a storage requirement for the specific file system by a number of the plurality of storage devices; and

redistributing a specific data set included in the specific file system across the plurality of storage devices based on the calculated target amounts of storage space.

6. A non-transitory computer readable storage medium including computer program instructions that, when executed, cause a computer processor to perform operations comprising:

maintaining a plurality of logically independent file systems, wherein each file system includes a data set stored by a distributed storage;

determining file system access pattern levels by observing accesses to data files in the data set included in each of the file systems;

determining which file system is most frequently accessed among the plurality of logically independent file systems;

establishing a dedicated physical partition in each of a plurality of disks for the data set of the most frequently accessed file system, the dedicated physical partition located near an outer edge of the respective disk;

and distributing the data set of the most frequently accessed file system across dedicated physical partitions of the plurality of disks before distributing data sets of other file systems of the plurality of logically independent file systems across another partition in each of the plurality of disks, wherein the data sets of the other file systems are distributed based at least in part on determined file system access pattern levels for the plurality of logically independent file systems;

determining that a specific file system from a plurality of other logically independent file systems has an access pattern level specifying a higher probability of future access than a probability of future access specified by an access pattern level of another one of the plurality of logically independent file systems; and

redistributing a data set included in the specific file system prior to redistributing a data set included in another file system based on the determination that the specific file system from the plurality of other logically independent file systems has the access pattern level specifying a higher probability of future access than the probability of future access specified by the access pattern level of the another one of the plurality of logically independent file systems.

7. The medium of claim 6 , wherein distributing the data sets of the other logically independent file systems comprises:

redistributing a particular data set included in a particular file system from the plurality of logically independent file systems uniformly across a plurality of storage devices of the distributed storage.

8. The medium of claim 6 , wherein distributing the data sets of the other logically independent file systems comprises:

determining storage requirements for the plurality of file systems;

obtaining device characteristic information for a plurality of storage devices of the distributed storage; and

redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage.

9. The medium of claim 8 , wherein redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage comprises:

redistributing a specific data set included in a specific file system from the plurality of logically independent file systems across the plurality of storage devices in proportion to a performance characteristic for the specific file system described by the performance characteristic information.

10. The medium of claim 8 , wherein redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage comprises:

calculating a target amount of storage space assigned to a specific file system for each of the plurality of storage devices by dividing a storage requirement for the specific file system by a number of the plurality of storage devices; and

redistributing a specific data set included in the specific file system across the plurality of storage devices based on the calculated target amounts of storage space.

11. A system comprising:

a hardware computer processor; and

a non-transitory computer readable storage medium storing processor-executable computer program instructions, the instructions comprising instructions for:

maintaining a plurality of logically independent file systems, wherein each file system includes a data set stored by a distributed storage;

determining file system access pattern levels by observing accesses to data files in the data set included in each of the file systems;

determining which file system is most frequently accessed among the plurality of logically independent file systems;

establishing a dedicated physical partition in each of a plurality of disks for the data set of the most frequently accessed file system, the dedicated physical partition located near an outer edge of the respective disk; and

distributing the data set of the most frequently accessed file system across the dedicated physical partitions of the plurality of disks before distributing the data sets of other file systems of the plurality of logically independent file systems across another partition in each of the plurality of disks, wherein the data sets of the other file systems are distributed based at least in part on determined file system access pattern levels for the plurality of logically independent file systems;

determining that a specific file system from a plurality of other logically independent file systems has an access pattern level specifying a higher probability of future access than a probability of future access specified by an access pattern level of another one of the plurality of logically independent file systems; and

redistributing a data set included in the specific file system prior to redistributing a data set included in another file system based on the determination that the specific file system from the plurality of other logically independent file systems has the access pattern level specifying a higher probability of future access than the probability of future access specified by the access pattern level of the another one of the plurality of logically independent file systems.

12. The system of claim 11 , wherein distributing the data sets of the other logically independent file systems comprises:

redistributing a particular data set included in a particular file system from the plurality of logically independent file systems uniformly across a plurality of storage devices of the distributed storage.

13. The system of claim 11 , wherein distributing the data sets of the other logically independent file systems comprises:

determining storage requirements for the plurality of file systems;

obtaining device characteristic information for a plurality of storage devices of the distributed storage; and

redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage.

14. The system of claim 11 , wherein redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage comprises:

redistributing a specific data set included in a specific file system from the plurality of logically independent file systems across the plurality of storage devices in proportion to a performance characteristic for the specific file system described by the performance characteristic information.

15. The system of claim 11 , wherein redistributing the data sets across the plurality of storage devices based on the storage requirements for the plurality of file systems and the obtained device characteristic information for the plurality of storage devices of the distributed storage comprises:

calculating a target amount of storage space assigned to a specific file system for each of the plurality of storage devices by dividing a storage requirement for the specific file system by a number of the plurality of storage devices;

and redistributing a specific data set included in the specific file system across the plurality of storage devices based on the calculated target amounts of storage space.

Assignments (9)

Continuity (1)