Data management system with limited control of external compute and storage resources
Methods and systems for improving data back-up, recovery, and search across different cloud-based applications, services, and platforms are described. A data management and storage system may direct compute and storage resources within a customer's cloud-based data storage account to back-up and restore data while the customer retains full control of their data. The data management and storage system may direct the compute and storage resources within the customer's cloud-based data storage account to generate and store secondary layers that are used for generating search indexes, to generate and store shared space layers and user specific layers to facilitate the deduplication of email attachments and text blocks, to perform a controlled restoration of email snapshots such that sensitive information (e.g., restricted keywords) located within stored snapshots remains protected, and to detect and preserve emails that were received or transmitted and then deleted between two consecutive snapshots.
1. A method for operating a data management system, comprising:
establishing a secure connection with a cloud-based storage account;
identifying storage resources corresponding to a physical disk space available within the cloud-based storage account and external to the data management system;
detecting that a first snapshot of a set of data is to be captured and stored within the cloud-based storage account;
directing the cloud-based storage account to acquire a first set of data associated with the first snapshot from a cloud-based service;
directing the cloud-based storage account to store the first snapshot using the storage resources, the first snapshot comprising a first set of fields;
determining a maximum file size for storing one or more search indexes based at least in part on an amount of available physical disk space within the storage resources and a frequency with which the first snapshot is updated at the cloud-based storage account;
determining that the maximum file size for storing one or more search indexes is less than a threshold disk space;
directing the cloud-based storage account to extract a subset of the first set of fields for the first snapshot and generate a first search index for the first snapshot based at least in part on determining that the maximum file size is less than the threshold disk space; and
directing the cloud-based storage account to store the first search index within the cloud-based storage account using the storage resources based at least in part on determining that a file size of the first search index is not greater than the maximum file size, wherein the first search index comprises a plurality of search keys for the set of data and a mapping between the plurality of search keys and a plurality of corresponding locations within the set of data.
2. The method of claim 1 , further comprising:
detecting a change in the amount of storage resources that are available within the cloud-based storage account; and
directing the cloud-based storage account to regenerate the first search index in response to detecting the change in the amount of the storage resources.
3. The method of claim 2 , wherein:
the detecting the change in the amount of the storage resources includes detecting a reduction in a total amount of physical disk space for the storage resources.
4. The method of claim 1 , further comprising:
detecting that a second snapshot of the set of data should be captured and stored within the cloud-based storage account;
directing the cloud-based storage account to acquire a second set of data associated with the second snapshot from the cloud-based service;
directing the cloud-based storage account to store the second snapshot using the storage resources;
determining a second file size for a second search index for the second snapshot; and
directing the cloud-based storage account to generate and store the second search index for the second snapshot using the storage resources such that the file size of the second search index is not greater than the second file size, the second file size is greater than the maximum file size.
5. The method of claim 1 , wherein:
the set of data comprises a plurality of electronic messages.
6. The method of claim 5 , wherein:
the first snapshot comprises a full image snapshot of the plurality of electronic messages at a first point in time.
7. The method of claim 1 , wherein:
the directing the cloud-based storage account to acquire the first set of data includes transmitting one or more instructions to the storage resources within the cloud-based storage account.
8. The method of claim 1 , wherein:
the establishing the secure connection with the cloud-based storage account includes establishing the secure connection using token-based authentication.
9. The method of claim 1 , wherein:
the first search index includes a word-level inverted index for the first snapshot.
10. The method of claim 1 , wherein:
the determining the first file size for the first search index for the first snapshot includes detecting that the amount of available physical disk space within the storage resources is greater than a threshold amount of physical disk space and setting the first file size in response to detecting that the amount of available physical disk space within the storage resources is greater than the threshold amount of physical disk space.
11. A data management system, comprising:
a memory configured to store an access token; and
one or more processors in communication with the memory configured to establish a secure connection with a cloud-based storage account using the access token and identify storage resources corresponding to a physical disk space available within the cloud-based storage account and external to the data management system, the one or more processors configured to detect that a first snapshot of a set of data is to be captured and stored within the cloud-based storage account and instruct the cloud-based storage account to acquire a first set of data associated with the first snapshot from a cloud-based service, the one or more processors configured to instruct the cloud-based storage account to store the first snapshot using the storage resources, the first snapshot comprising a first set of fields, determine a maximum file size for storing one or more search indexes based at least in part on an amount of available physical disk space for the storage resources and a frequency with which the first snapshot is updated at the cloud-based storage account, and determine that the maximum file size for storing one or more search indexes is less than a threshold disk space, the one or more processors configured to instruct the cloud-based storage account to extract a subset of the first set of fields for the first snapshot and generate a first search index for the first snapshot based at least in part on determining that the maximum file size is less than the threshold disk space, and the one or more processors are further configured to instruct the cloud-based storage account to store the first search index within the cloud-based storage account using the storage resources based at least in part on determining that a file size of the first search index is not greater than the maximum file size, wherein the first search index comprises a plurality of search keys for the set of data and a mapping between the plurality of search keys and a plurality of corresponding locations within the set of data.
12. The data management system of claim 11 , wherein:
the one or more processors configured to detect a change in the amount of storage resources that are available within the cloud-based storage account and direct the cloud-based storage account to regenerate the first search index in response to detection of the change in the amount of the storage resources.
13. The data management system of claim 12 , wherein:
the one or more processors configured to detect a reduction in a total amount of physical disk space for the storage resources and direct the cloud-based storage account to regenerate the first search index in response to detection of the reduction in the total amount of physical disk space for the storage resources.
14. The data management system of claim 11 , wherein:
the one or more processors configured to detect that a second snapshot of the set of data should be captured and stored within the cloud-based storage account and direct the cloud-based storage account to acquire a second set of data associated with the second snapshot from the cloud-based service, the one or more processors configured to direct the cloud-based storage account to store the second snapshot using the storage resources and determine a second file size for a second search index for the second snapshot, the one or more processors configured to direct the cloud-based storage account to generate and store the second search index for the second snapshot using the storage resources such that the file size of the second search index is not greater than the second file size, the second file size is greater than the maximum file size.
15. The data management system of claim 11 , wherein: the set of data comprises a plurality of electronic messages.
16. The data management system of claim 15 , wherein:
the first snapshot comprises a full image snapshot of the plurality of electronic messages at a first point in time.
17. The data management system of claim 11 , wherein:
the one or more processors configured to direct the cloud-based storage account to acquire the first set of data via transmission of one or more instructions to the storage resources within the cloud-based storage account.
18. The data management system of claim 11 , wherein:
the first search index includes a word-level inverted index for the first snapshot.
19. The data management system of claim 11 , wherein:
the set of data comprises an electronic message; and
the first snapshot comprises an incremental snapshot of the electronic message at a first point in time.
20. One or more storage devices containing processor readable code for programming one or more processors to perform a method for operating a data management system, the processor readable code comprising:
processor readable code configured to establish a secure connection with a cloud-based storage account;
processor readable code configured to identify storage resources corresponding to a physical disk space available within the cloud-based storage account and external to the data management system;
processor readable code configured to detect that a first snapshot of a set of data is to be captured and stored within the cloud-based storage account;
processor readable code configured to instruct the cloud-based storage account to acquire a first set of data associated with the first snapshot from a cloud-based service;
processor readable code configured to instruct the cloud-based storage account to store the first snapshot using the storage resources, the first snapshot comprising a first set of fields;
processor readable code configured to determine a maximum file size for storing one or more search indexes based at least in part on an amount of available physical disk space within the storage resources and a frequency with which the first snapshot is updated at the cloud-based storage account;
processor readable code configured to determine that the maximum file size for storing one or more search indexes is less than a threshold disk space;
processor readable code configured to instruct the cloud-based storage account to extract a subset of the first set of fields for the first snapshot and generate a first search index for the first snapshot based at least in part on determining that the maximum file size is less than the threshold disk space; and
processor readable code configured to instruct the cloud-based storage account to store a first search index within the cloud-based storage account using the storage resources based at least in part on determining that a file size of the first search index is not greater than the maximum file size, wherein the first search index comprises a plurality of search keys for the set of data and a mapping between the plurality of search keys and a plurality of corresponding locations within the set of data.