Adaptable multi-layered storage for deduplicating electronic messages
Methods and systems for improving data back-up, recovery, and search across different cloud-based applications, services, and platforms are described. A data management and storage system may direct compute and storage resources within a customer's cloud-based data storage account to back-up and restore data while the customer retains full control of their data. The data management and storage system may direct the compute and storage resources within the customer's cloud-based data storage account to generate and store secondary layers that are used for generating search indexes, to generate and store shared space layers and user specific layers to facilitate the deduplication of email attachments and text blocks, to perform a controlled restoration of email snapshots such that sensitive information (e.g., restricted keywords) located within stored snapshots remains protected, and to detect and preserve emails that were received or transmitted and then deleted between two consecutive snapshots.
1. A method, comprising:
instantiating an archive application in connection with an account hosted by a cloud-based data storage service;
periodically accessing, using the archive application, one or more accounts hosted by a cloud-based electronic messaging service to retrieve and store, in the cloud-based data storage service, one or more snapshots, the cloud-based electronic messaging service storing electronic messages associated with a plurality of user messaging accounts, each snapshot comprising a plurality of electronic messages associated with the plurality of user messaging accounts at a corresponding point in time;
for at least a first snapshot of the one or more snapshots:
parsing the electronic messages to create one or more searchable indexes based on one or more attributes of the electronic messages; and
storing the one or more searchable indexes in the cloud-based data storage service; and
for two or more electronic messages in the first snapshot:
identifying an attachment common to the two or more electronic messages;
detecting that the attachment is to be stored within a shared space;
storing the attachment within the shared space of the cloud-based data storage service; and
storing pointers to the attachment in association with the two or more electronic messages.
2. The method of claim 1 , wherein the archive application is hosted on resources of the cloud-based data storage service.
3. The method of claim 1 , wherein access to the cloud-based data storage service is restricted.
4. The method of claim 1 , wherein one or more operations of the archive application are orchestrated by a data backup and index generation controller that is remote from the archive application.
5. The method of claim 4 , wherein the data backup and index generation controller is operative to establish a secure connection to the cloud-based data storage service.
6. The method of claim 1 , wherein identifying the attachment common to the two or more electronic messages comprises:
comparing one or more hash values generated from a first attachment of a first electronic message to one or more hash values from a second attachment of at least a second electronic message.
7. The method of claim 1 , wherein:
at least a first electronic message of the two or more electronic messages is acquired from a first email mailbox; and
a second electronic message of the two or more electronic messages is acquired from the first email mailbox.
8. The method of claim 1 , wherein:
at least a first electronic message of the two or more electronic messages is acquired from a first email mailbox; and
a second electronic message of the two or more electronic messages is acquired from a second email mailbox different from the first email mailbox.
9. The method of claim 1 , wherein detecting that the attachment is to be stored in the shared space comprises:
identifying a text block within an electronic message of the two or more electronic messages;
determining an aggregate data size for the text block based on a number of recipients of the electronic message; and
detecting that the text block is to be stored within the shared space based on the aggregate data size for the text block.
10. One or more storage devices containing processor-readable code for programming one or more processors to perform a method for operating a data management system, the processor-readable code comprising:
processor-readable code configured to instantiate an archive application in connection with an account hosted by a cloud-based data storage service;
processor-readable code configured to periodically access, using the archive application, one or more accounts hosted by a cloud-based electronic messaging service to retrieve and store, in the cloud-based data storage service, one or more snapshots, the cloud-based electronic messaging service storing electronic messages associated with a plurality of user messaging accounts, each snapshot comprising a plurality of electronic messages associated with the plurality of user messaging accounts at a corresponding point in time;
processor-readable code configured to, for at least a first snapshot of the one or more snapshots:
parse the electronic messages to create one or more searchable indexes based on one or more attributes of the electronic messages; and
store the one or more searchable indexes in the cloud-based data storage service; and
processor-readable code configured to, for two or more electronic messages in the first snapshot:
identify an attachment common to the two or more electronic messages;
detect that the attachment is to be stored within a shared space;
store the attachment within the shared space of the cloud-based data storage service; and
store pointers to the attachment in association with the two or more electronic messages.
11. The one or more storage devices of claim 10 , wherein the archive application is configured to be hosted on resources of the cloud-based data storage service.
12. The one or more storage devices of claim 10 , wherein access to the cloud-based data storage service is restricted.
13. The one or more storage devices of claim 10 , wherein one or more operations of the archive application are configured to be orchestrated by a data backup and index generation controller that is remote from the archive application.
14. The one or more storage devices of claim 13 , wherein the data backup and index generation controller is operative to establish a secure connection to the cloud-based data storage service.
15. The one or more storage devices of claim 10 , wherein the processor-readable code configured to identify the attachment common to the two or more electronic messages comprises processor-readable code configured to:
compare one or more hash values generated from a first attachment of a first electronic message of the two or more electronic messages to one or more hash values from a second attachment of a second electronic message of the two or more electronic messages.
16. The one or more storage devices of claim 10 , wherein:
at least a first electronic message of the two or more electronic messages is associated with a first email mailbox; and
a second electronic message of the two or more electronic messages is associated with the first email mailbox.
17. The one or more storage devices of claim 10 , wherein:
at least a first electronic message of the two or more electronic messages is associated with a first email mailbox; and
a second electronic message of the two or more electronic messages is associated with a second email mailbox that is different from the first email mailbox.
18. The one or more storage devices of claim 10 , wherein the processor-readable code configured to detect that the attachment is to be stored in the shared space comprises processor-readable code configured to:
identify a text block within an electronic message of the two or more electronic messages;
determine an aggregate data size for the text block based on a number of recipients of the electronic message; and
detect that the text block is to be stored within the shared space based on the aggregate data size for the text block.
19. The one or more storage devices of claim 10 , wherein the processor-readable code configured to detect that the attachment is to be stored in the shared space comprises processor-readable code configured to:
detect that at least a first message of the two or more electronic messages was sent to more than a threshold a number of email addresses.
20. The one or more storage devices of claim 10 , wherein the processor-readable code configured to detect that the attachment is to be stored in the shared space comprises processor-readable code configured to:
determine an aggregate file size of the two or more electronic messages.