Efficient data storage management system
An efficient data storage system is described. An agent software application on computing devices in a first tier processes snapshot backups and pushes them to an appliance software application on a server in a second tier. The appliance software application processes archive backups and pushes them to cloud storage in a third tier. A cloud application on a management server receives storage policy specifications from customers and promulgates the policies to the agent software application and the appliance software application. The policy specifications include time periods and retention set information for the backups. When a retention set has been exceeded, the storage system is pruned to remove file references to unneeded files and delete data files no longer referenced in storage sets in the retention set.
1. A method for pruning data in an efficient data storage system comprising:
in a maintenance set of data comprising file references and data files, evaluating each file in the maintenance set in view of an oldest storage set of data in a retention set, the evaluating including performing for each file in the maintenance set of data the following actions for a current file in the maintenance set of data
evaluating whether a kind of the current file is file reference or data file,
when the kind of the current file is file reference,
deleting the current file from the maintenance set, the data storage system thus recapturing the file reference;
when the kind of the current file is data file,
checking whether the current file is referenced in an oldest storage set of data in the retention set by checking each file reference in the oldest storage set to learn whether the file reference refers to the current file
when there is no reference to the current file in the oldest storage set of data in the retention set, deleting the data file from the maintenance set of data, the data storage system thus recapturing the data file.
2. A data storage system comprising a computer server including a processor and software which when executed by the processor causes the computer server to perform actions including:
evaluating each file in a maintenance set of data in comparison to an oldest storage set of data in a retention set, the evaluating including performing for each file in the maintenance set of data the following actions for a current file in the maintenance set of data
evaluating whether a kind of the current file is file reference or data file,
when the kind of the current file is file reference,
deleting the current file from the maintenance set, thus recapturing the file reference;
when the kind of the current file is data file,
checking whether the current file is referenced in the oldest storage set of data in the retention set by checking each file reference in the oldest storage set to learn whether the file reference refers to the current file
when there is no reference to the current file in the oldest storage set of data in the retention set, deleting the current file from the maintenance set of data, thus recapturing the data file.
3. The method of claim 1 :
wherein each of the plurality of storage sets of data include a plurality of data files and a plurality of file references.
wherein the maintenance set of data includes storage sets that preceded the retention set.
wherein the retention set includes a number of storage sets retained by the data storage system according to a policy.
4. The data storage system of claim 2 :
wherein each of the plurality of storage sets of data include a plurality of data files and a plurality of file references.
wherein the maintenance set of data includes storage sets that preceded the retention set.
wherein the retention set includes a number of storage sets retained by the data storage system according to a policy.