Preservation of electronic messages between snapshots
Methods and systems for improving data back-up, recovery, and search across different cloud-based applications, services, and platforms are described. A data management and storage system may direct compute and storage resources within a customer's cloud-based data storage account to back-up and restore data while the customer retains full control of their data. The data management and storage system may direct the compute and storage resources within the customer's cloud-based data storage account to generate and store secondary layers that are used for generating search indexes, to generate and store shared space layers and user specific layers to facilitate the deduplication of email attachments and text blocks, to perform a controlled restoration of email snapshots such that sensitive information (e.g., restricted keywords) located within stored snapshots remains protected, and to detect and preserve emails that were received or transmitted and then deleted between two consecutive snapshots.
1. A method for operating a data management system, comprising:
acquiring a first snapshot of a set of electronic messages, the first snapshot comprises a state of the set of electronic messages at a first point in time;
detecting that a first electronic message was received subsequent to the first point in time;
buffering the first electronic message using a memory buffer in response to detecting that the first electronic message was received subsequent to the first point in time;
acquiring a second snapshot of the set of electronic messages subsequent to buffering the first electronic message using the memory buffer, the second snapshot comprises a state of the set of electronic messages at a second point in time subsequent to the first point in time;
comparing electronic messages stored within the memory buffer with electronic messages within the second snapshot of the set of electronic messages;
detecting that the second snapshot of the set of electronic messages does not include the first electronic message based on the comparison of the electronic messages stored within the memory buffer with the electronic messages within the second snapshot of the set of electronic messages;
storing the first electronic message within a buffer snapshot in response to detecting that the second snapshot of the set of electronic messages does not include the first electronic message; and
deleting the first electronic message from the memory buffer subsequent to storing the first electronic message within the buffer snapshot.
2. The method of claim 1 , further comprising:
detecting that the first electronic message includes a restricted keyword; and
removing the first electronic message from the buffer snapshot in response to detecting that the first electronic message includes the restricted keyword.
3. The method of claim 1 , further comprising:
detecting that the first electronic message includes a restricted keyword; and
storing the first electronic message within the buffer snapshot in response to detecting that the first electronic message includes the restricted keyword and detecting that the second snapshot of the set of electronic messages does not include the first electronic message.
4. The method of claim 1 , further comprising:
detecting that the first electronic message does not include a restricted keyword; and
storing the first electronic message within the buffer snapshot in response to detecting that the first electronic message does not include the restricted keyword.
5. The method of claim 1 , further comprising:
detecting that the first electronic message has been searched within a past threshold period of time; and
storing the first electronic message within the buffer snapshot in response to detecting that the first electronic message has been searched within the past threshold period of time.
6. The method of claim 1 , wherein:
the detecting that the first electronic message was received subsequent to the first point in time includes intercepting the first electronic message as an inbound email message.
7. The method of claim 1 ; further comprising:
detecting that a second electronic message was transmitted subsequent to the first point in time;
buffering the second electronic message using the memory buffer;
detecting that the second snapshot of the set of electronic messages does not include the first electronic message or the second electronic message;
storing the first electronic message in the buffer snapshot in response to detecting that the second snapshot does not include the first electronic message; and
storing the second electronic message in the buffer snapshot in response to detecting that the second snapshot does not include the second electronic message.
8. The method of claim 1 , wherein:
the buffering the first electronic message comprises buffering the first electronic message within a cloud-based storage account.
9. The method of claim 1 , wherein:
the set of electronic messages includes a first set of email messages associated with a first user's email mailbox and a second set of email messages associated with a second user's email mailbox.
10. The method of claim 1 , wherein:
the set of electronic messages corresponds with email messages across a plurality of email mailboxes; and
the time difference between the first point in time and the second point in time comprises an hour.
11. A data management system, comprising:
a memory buffer; and
one or more processors in communication with the memory buffer configured to acquire a first snapshot of a set of electronic messages, the first snapshot comprises a state of the set of electronic messages at a first point in time, the one or more processors configured to detect that a first electronic message was received subsequent to the first point in time and write the first electronic message to the memory buffer, the one or more processors configured to acquire a second snapshot of the set of electronic messages subsequent to writing the first electronic message to the memory buffer, the second snapshot comprises a state of the set of electronic messages at a second point in time subsequent to the first point in time, the one or more processors configured to compare electronic messages stored within the memory buffer with electronic messages within the second snapshot of the set of electronic messages, the one or more processors configured to detect that the second snapshot of the set of electronic messages does not include the first electronic message based on the comparison of the electronic messages stored within the memory buffer with the electronic messages within the second snapshot of the set of electronic messages, the one or more processors configured to store the first electronic message within a buffer snapshot in response to detection that the second snapshot of the set of electronic messages does not include the first electronic message and delete the first electronic message from the memory buffer subsequent to the first electronic message being stored within the buffer snapshot.
12. The data management system of claim 11 , wherein:
the one or more processors configured to detect that the first electronic message includes a restricted keyword and remove the first electronic message from the buffer snapshot in response to detection that the first electronic message includes the restricted keyword.
13. The data management system of claim 11 , wherein:
the one or more processors configured to detect that the first electronic message does not include a restricted keyword and remove the first electronic message from the buffer snapshot in response to detection that the first electronic message does not include the restricted keyword.
14. The data management system of claim 11 , wherein:
the one or more processors configured to detect that the first electronic message has been searched within a past threshold period of time and store the first electronic message within the buffer snapshot in response to detection that the first electronic message has been searched within the past threshold period of time and detection that the second snapshot of the set of electronic messages does not include the first electronic message.
15. The data management system of claim 11 , wherein:
the one or more processors configured to intercept the first electronic message as an inbound email message.
16. The data management system of claim 11 , wherein:
the one or more processors configured to detect that a second electronic message was transmitted subsequent to the first point in time and write the second electronic message to the memory buffer, the one or more processors configured to detect that the second snapshot of the set of electronic messages does not include the first electronic message or the second electronic message, the one or more processors configured to store the first electronic message and the second electronic message in the buffer snapshot in response to detection that the second snapshot does not include either the first electronic message or the second electronic message.
17. The data management system of claim 11 , wherein:
the buffer snapshot is stored within a cloud-based data storage account; and
the memory buffer comprises a volatile memory buffer.
18. The data management system of claim 11 , wherein:
the set of electronic messages includes a first set of email messages associated with a first user's email mailbox and a second set of email messages associated with a second user's email mailbox.
19. The data management system of claim 11 , wherein:
the set of electronic messages corresponds with a plurality of email messages across a plurality of email mailboxes.
20. One or more storage devices containing processor readable code for programming one or more processors to perform a method for operating a data management system, the processor readable code comprising:
processor readable code configured to acquire a first snapshot of a set of electronic messages, the first snapshot comprises a state of the set of electronic messages at a first point in time;
processor readable code configured to detect that a first electronic message was received subsequent to the first point in time;
processor readable code configured to buffer the first electronic message using a memory buffer in response to detection that the first electronic message was received subsequent to the first point in time;
processor readable code configured to acquire a second snapshot of the set of electronic messages subsequent to buffering the first electronic message using the memory buffer, the second snapshot comprises a state of the set of electronic messages at a second point in time subsequent to the first point in time;
processor readable code configured to compare electronic messages stored within the memory buffer with electronic messages within the second snapshot of the set of electronic messages;
processor readable code configured to detect that the second snapshot of the set of electronic messages does not include the first electronic message based on the comparison of the electronic messages stored within the memory buffer with the electronic messages within the second snapshot of the set of electronic messages;
processor readable code configured to store the first electronic message within a buffer snapshot in response to detection that the second snapshot of the set of electronic messages does not include the first electronic message; and
processor readable code configured to delete the first electronic message from the memory buffer subsequent to storing the first electronic message within the buffer snapshot.