De-duplicating attachments on message delivery and automated repair of attachments
Systems and techniques of de-duplicating file and/or blobs within a file system are presented. In one embodiment, an email system is disclosed wherein the email system receives email messages comprising a set of associated attachments. The system determines whether the associated attachments have been previously stored in the email system, the state of the stored attachment, and if the state of the attachment is appropriate for sharing copies of the attachment, then providing a reference to the attachment upon a request to share the attachment. In another embodiment, the system may detect whether stored attachments are corrupted and, if so, attempt to repair the attachment, and possibly, prior to sharing references to the attachment.
1. A method for de-duplicating data in a file system, said file system comprising one or more files, each said file comprising a set of data and said set of data capable of being shared by two or more users, the steps of said method comprising:
receiving a file to be stored within a file system;
checking whether a set of data within said file has been previously stored in said file system;
if said set of data has been stored in said file system, checking the state of the stored set of data;
if said state of stored set of data is appropriate for sharing copies of said file, then
providing a reference to said file upon a command to share said stored set of data;
incrementing a reference count associated with said stored set of data;
checking the reference count; and
if the reference count is zero, transition the state of said stored set of data to a state that is one of a group, said group consisting of: deleting, tentative, orphan and recovered.
2. The method of claim 1 wherein said file system is one of a group, said group comprising: a database, a file system and an email system.
3. The method of claim 1 wherein said file system is an email system, said files comprise emails and said set of data comprises attachments to said emails.
4. The method of claim 3 wherein said step of receiving a file to be stored within a file system further comprises:
receiving an email, said email further comprising an attachment.
5. The method of claim 4 wherein said step of receiving a file to be stored within a file system further comprises:
storing metadata about said email, said metadata comprising one of a group, said group comprising: metadata regarding the email, metadata regarding an attachment to the email, state metadata regarding the email, state metadata regarding the attachment, metadata regarding how to share the attachment and metadata regarding duplication of said attachment.
6. The method of claim 4 wherein said step of checking whether a set of data within said file has been previously stored in said file system further comprises:
performing a hash function on said set of data; and
comparing the results of said hash function with results of previous hash results.
7. The method of claim 4 wherein said step of checking the state of the stored set of data further comprises:
assigning a state associated with said attachment.
8. The method of claim 7 wherein said states associated with said attachment comprise one of a group, said group comprising: Good, SomeCorrupt, AllCorrupt, Tentative, Deleting, Orphan and Recovered.
9. The method of claim 7 wherein said step of: if said state of stored set of data is appropriate for sharing copies of said file, then providing a reference to said file upon a command to share said set of data further comprises:
incrementing a reference count for said stored set of data; and
sending to the target entity a pointer reference to said stored set of data.
10. The method of claim 1 wherein said method further comprises the step of:
detecting if a set of data is corrupted; and
if corrupted, transitioning the state associated with said set of data from good to corrupt.
11. The method of claim 10 wherein said method further comprises the step of:
repairing said corrupt set of data; and
transitioning the state associated with said set of data from corrupt to good.
12. A system for de-duplicating data in an email system, said email system comprising:
an email server, said email server in communication with a set of email clients;
a database for storing emails and associated attachments; said database storing at least one of group, said group comprising: the email message, associated attachments, metadata regarding the email message and metadata regarding said associated attachments;
a module for managing the sharing of attachments between said set of email clients;
wherein said module of managing the sharing of attachments between said set of email clients is capable of detecting the state of an attachment and sharing a reference pointer to said attachment depending upon the state of said attachment; and
further wherein said module of managing the sharing of attachments between said set of email clients is capable of:
checking a reference count associated with said attachment;
if the attachment is shared, incrementing the reference count; and
if the reference count is zero, transitioning the state of the attachment to one of a group, said group consisting of: deleting, tentative, orphan and recovered.
13. The system of claim 12 wherein said module for managing the sharing of attachments further comprises a module for detecting whether an attachment is corrupt.
14. The system of claim 13 wherein said module for managing the sharing of attachments further comprises a module for maintaining the state of an attachment.
15. The system of claim 14 wherein said module for managing the sharing of attachments further comprises a module for repairing a corrupt attachment.
16. The system of claim 15 wherein said module for managing the sharing of attachments further comprises a module for removing attachments that are no longer referenced by email clients.
17. The system of claim 16 wherein said module for managing the sharing of attachments further comprises a module for transition the state of an attachment depending upon the detected state of said attachment.
18. A computer readable I/O storage medium, said computer readable storage medium having computer-executable instructions stored thereon that, when executed by a processor, cause said processor to execute: a method for de-duplicating data in a file system, said file system comprising one or more files, each said file comprising a set of data and said set of data capable of being shared by two or more users, the steps of said method comprising:
receiving a file to be stored within a file system;
checking whether a set of data within said file has been previously stored in said file system;
if said set of data has been stored in said file system, checking the state of the stored set of data; and
if said state of stored set of data is appropriate for sharing copies of said file, then
providing a reference to said file upon a command to share said set of data;
incrementing a reference count associated with said stored set of data;
checking the reference count; and
if the reference count is zero, transition the state of said stored set of data to a state that is one of a group, said group comprising: deleting, tentative, orphan and recovered.
19. The computer readable storage medium of claim 18 wherein said step of checking whether a set of data within said file has been previously stored in said file system further comprises:
performing a hash function on said set of data; and
comparing the results of said hash function with results of previous hash results.
20. The computer readable medium of claim 19 wherein said step of:
if said state of stored set of data is appropriate for sharing copies of said file, then providing a reference to said file upon a command to share said set of data further comprises:
incrementing a reference count for said stored set of data; and
sending to the target entity a pointer reference to said stored set of data.