Information source agent systems and methods for distributed data storage and management using content signatures
Information source agent systems and methods for distributed content storage and management using content signatures that use file identicality properties are provided. A data management system is provided that includes a content engine for managing the storage of file content, a content signature generator that generates a unique content signature for a file processed by the content engine, a content signature comparator that compares content signatures and a content signature repository that stores content signatures. Information source agents are provided that include content signature generators and content signature comparators. Methods are provided for the efficient management of files using content signatures that take advantage of file identicality properties. Content signature application modules and registries exist within information source clients and centralized servers to support the content signature methods.
1. A method comprising:
generating a file content index for a file received at a computing device from an information source client;
comparing the file content index to stored file content indices to determine a similarity between the file content index and the stored file content indices and to determine whether the similarity exceeds a similarity threshold;
comparing, if the similarity exceeds the similarity threshold, differences between the received file and files that have the file content indices that exceed the similarity threshold;
determining, from among the files that have the file content indices that exceed the similarity threshold, a closest match file to the received file; and
creating a delta file of the differences between the received file and the closest match file.
2. The method of claim 1 , further comprising:
updating a file identifier for the closest match file to account for the differences;
storing the content of the received file; and
storing the file content index,
wherein the file identifier comprises a unique value generated from the received file.
3. The method of claim 1 , further comprising:
storing the delta file if a storage factor is above a threshold value.
4. The method of claim 1 , further comprising:
storing at least one of the delta file, the received tile, and the file content index if a storage factor is below a threshold value.
5. The method of claim 4 , wherein the threshold is a storage threshold of an indexed archive system.
6. The method of claim 4 , wherein the threshold is based on file type.
7. The method of claim 4 , wherein the threshold is based on level of interest of a characteristic of the received file.
8. The method of claim 1 , further comprising:
determining whether previously received versions of the received file have been previously indexed; and
storing a link to map the previously received versions of the received file to the received file.
9. The method of claim 1 , further comprising:
storing a link indicating the received file shares content indices exceeding a similarity threshold for at least one stored file.
10. An non-transitory computer readable medium having stored thereon in digital form computer-executable instructions that, in response to execution by a computing device, cause the computing device to perform operations for storing file information in an indexed archive system, the operations comprising:
generating a file content index for a file received at the computing device from an information source client;
comparing the file content index to stored file content indices to determine a similarity between the file content index and the stored file content indices and to determine whether the similarity exceeds a similarity threshold;
comparing, if the similarity exceeds the similarity threshold, differences between the received file and files that have the file content indices that exceed the similarity threshold;
determining, from among the files that have the file content indices that exceed the similarity threshold, a closest match file to the received file; and
creating a delta file of the differences between the received file and the closest match file.
11. The non-transitory computer readable medium of claim 10 , the operations further comprising:
updating a file identifier for the closest match file to account for the differences;
storing the content of the received file; and
storing the file content index,
wherein the file identifier comprises a unique value generated from the received file.
12. The non-transitory computer readable medium of claim 10 , the operations further comprising:
storing the delta file if a storage factor is above a threshold value.
13. The non-transitory computer readable medium of claim 10 , the operations further comprising:
storing at least one of the delta file, the received file, and the file content index if a storage factor is below a threshold value.
14. The non-transitory computer readable medium of claim 13 , wherein the threshold is a storage threshold of the indexed archive system.
15. The non-transitory computer readable medium of claim 13 , wherein the threshold is based on file type.
16. The non-transitory computer readable medium of claim 13 , wherein the threshold is based on level of interest of a characteristic of the received file.
17. The non-transitory computer readable medium of claim 10 , the operations further comprising:
determining whether previously received versions of the received file have been previously indexed; and
storing a link to map the previously received versions of the received file to the received file.
18. The non-transitory computer readable medium of claim 10 , the operations further comprising:
storing a link indicating the received file shares content indices exceeding a similarity threshold for at least one stored file.