Reducing input/output (I/O) operations for centralized backup and storage
View Patent ↗Techniques are described for reducing I/O operations and storage capacity requirements for centralized backup storage systems. A central server optimizes the collection and centralization of backup data from a number of endpoint devices for backup purposes. The central server utilizes a single instance store and a persistent files cache to minimize the number of backup copies for each non-unique file, reduce storage usage, network traffic, memory footprint and CPU cycles required to identify and process non-unique data. For each file in the single instance store, the server tracks the source device of that file until a threshold number of devices have been reached. Once the file reaches the threshold number of sources, the file is marked as persistent and its hash value is placed in the persistent files cache. Thereafter, all pointer creation and reference counting for that file cease.
1. A method for reducing input/output (I/O) operations for centralized data storage, the method comprising:
receiving, from each of a plurality of endpoint devices to a central server, a manifest that identifies a listing of files located on said each endpoint device;
for each received manifest, inspecting the manifest received to the central server to determine which files need to be uploaded from the endpoint device to the central server in order to construct a full image of the endpoint device on the central server, the inspecting performed by:
for each file identified in the manifest received from an endpoint device,
determining whether the file is available on the central server and requesting the file if the file is not available on the central server;
if the file is available on the central server, determining whether the file has been marked as persistent;
if the file has not been marked as persistent, storing an indication that the endpoint device is a source of the file; and
determining whether the file has at least a threshold number of source endpoint devices and marking the file as persistent if the file has at least the threshold number of source endpoint devices.
2. The method of claim 1 , further comprising:
storing a hash value associated with the file into a persistent files cache on the central server once the file has been marked as persistent.
3. The method of claim 1 , wherein determining that the file has been marked as persistent further comprises:
if the file has been marked as persistent, discontinuing the storing of indications that the endpoint device is the source of the file for all subsequently received manifests that identify the file.
4. The method of claim 1 , further comprising:
pre-calculating hash values for system files of an operating system and storing the hash values in the persistent files cache.
5. The method of claim 1 , wherein all files marked as persistent are restricted from being deleted.
6. The method of claim 1 , wherein the file is stored in a single instance store, such that only a single copy of each unique file is stored in the single instance store across all endpoint devices.
7. The method of claim 1 , wherein determining whether the file has at least a threshold number of source endpoint devices further comprises:
if the file does not have at least the threshold number of source endpoint devices, creating a pointer that associates the file with a centralized virtual disk of the endpoint device that is stored on the central server.
8. A central server computing device, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the computing device to:
receive a manifest from each of a plurality of endpoint devices, the manifest identifying files on the corresponding endpoint device; and
for each file identified in the manifest received from an endpoint device,
determine whether the file is available on the central server and request the file if the file is not available on the central server; and
if the file is available on the central server, determine whether the file has been marked as persistent;
if the file has not been marked as persistent, store an indication that the endpoint device is a source of the file; and
determine whether the file has at least a threshold number of source endpoint devices and mark the file as persistent if the file has at least the threshold number of source endpoint devices.
9. The central server computing device of claim 8 , the memory further comprising instructions to cause the processor to:
store a hash value associated with the file into a persistent files cache on the central server once the file has been marked as persistent.
10. The central server computing device of claim 8 , wherein determining that the file has been marked as persistent further comprises:
if the file has been marked as persistent, discontinuing the storing of indications that the endpoint device is the source of the file for all subsequently received manifests that identify the file.
11. The central server computing device of claim 8 , the memory further comprising instructions to cause the processor to:
pre-calculating hash values for system files of an operating system and storing the hash values in the persistent files cache.
12. The central server computing device of claim 8 , wherein all files marked as persistent are restricted from being deleted.
13. The central server computing device of claim 8 , wherein the file is stored in a single instance store, such that only a single copy of each unique file is stored in the single instance store across all endpoint devices.
14. The central server computing device of claim 8 , wherein determining whether the file has at least a threshold number of source endpoint devices further comprises:
if the file does not have at least the threshold number of source endpoint devices, creating a pointer that associates the file with a centralized virtual disk of the endpoint device that is stored on the central server.
15. A non-transitory computer readable storage medium comprising one or more sequences of instructions, the instructions when executed by one or more processors causing the one or more processors to execute the operations of:
receiving, from each of a plurality of endpoint devices to a central server, a manifest that identifies a listing of files located on said each endpoint device;
for each received manifest, inspecting the manifest received to the central server to determine which files need to be uploaded from the endpoint device to the central server in order to construct a full image of the endpoint device on the central server, the inspecting performed by:
for each file identified in the manifest received from an endpoint device,
determining whether the file is available on the central server and requesting the file if the file is not available on the central server; and
if the file is available on the central server, determining whether the file has been marked as persistent;
if the file has not been marked as persistent, storing an indication that the endpoint device is a source of the file; and
determining whether the file has at least a threshold number of source endpoint devices and marking the file as persistent if the file has at least the threshold number of source endpoint devices.
16. The non-transitory computer readable storage medium of claim 15 , further comprising instructions for:
storing a hash value associated with the file into a persistent files cache on the central server once the file has been marked as persistent.
17. The non-transitory computer readable storage medium of claim 15 , wherein determining that the file has been marked as persistent further comprises:
if the file has been marked as persistent, discontinuing the storing of indications that the endpoint device is the source of the file for all subsequently received manifests that identify the file.
18. The non-transitory computer readable storage medium of claim 15 , further comprising:
pre-calculating hash values for system files of an operating system and storing the hash values in the persistent files cache.
19. The non-transitory computer readable storage medium of claim 15 , wherein all files marked as persistent are restricted from being deleted.
20. The non-transitory computer readable storage medium of claim 15 , wherein the file is stored in a single instance store, such that only a single copy of each unique file is stored in the single instance store across all endpoint devices.