Data object store and server for a cloud storage environment, including data deduplication and data management across multiple cloud storage sites
Data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, are performed within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.
1. A computer-implemented method for encrypting files stored within a cloud storage site, the computer-implemented method comprising:
receiving a request to encrypt a data file located on a cloud storage site;
receiving an indication of a requested type of encryption method to utilize when encrypting the data file located on the cloud storage site,
wherein the requested type of encryption method is one of multiple types of encryption methods; and
determining whether the requested type of encryption method is supported by one or more application programming interfaces (APIs) provided by the cloud storage site;
when the requested type of encryption method is supported by the one or more APIs provided by the cloud storage site,
then utilizing the one or more APIs provided by the cloud storage site to encrypt the data file; and
when the requested type of encryption method is not supported by the one or more APIs provided by the cloud storage site,
then performing:
downloading a copy of the data file from the cloud storage site;
content indexing data in the downloaded copy of the data file by creating or updating at least one index associated with the data file;
encrypting the downloaded copy of the data file to create an encrypted version of the data file; and
sending the encrypted version of the data file to the cloud storage site via the one or more APIs to either—
overwrite the data file stored on the cloud storage site with the encrypted version of the data file, or
write the encrypted version of the data file to the cloud storage site and delete the data file stored on the cloud storage site.
2. The computer-implemented method of claim 1 , the computer-implemented method further comprising:
when the requested type of encryption method is supported by the one or more APIs provided by the cloud storage site, then utilizing a mapping to generate API calls to encrypt the data file.
3. The computer-implemented method of claim 1 , the computer-implemented method further comprising receiving one or more encryption keys from a user computer associated with the data file.
4. The computer-implemented method of claim 1 , the computer-implemented method further comprising transmitting APIs to the cloud storage site using at least one of hypertext transfer protocol (HTTP) and HTTP over Transport Layer Security/Secure Sockets Layer.
5. The computer-implemented method of claim 1 , the computer-implemented method further comprising providing a web-based search capability for searching for content in the data file, wherein the web-based search capability is performed without having to access data stored in the cloud storage site.
6. The computer-implemented method of claim 1 , the computer-implemented method further comprising receiving a request to compress a data file located on a cloud storage site, and determining whether the requested compression is supported by one or more application programming interfaces (APIs) provided by the cloud storage site.
7. The computer-implemented method of claim 1 , the computer-implemented method further comprising using a cloud storage site API to provide a command to encrypt the data file using an encryption method that does not result in the cloud storage site receiving a decryption key or does not result in the cloud storage site retaining information sufficient to decrypt the data file.
8. A computer-implemented method for pruning files containing data, the computer-implemented method comprising:
receiving an indication to delete a first file, wherein the first file includes a first set of data, and
wherein the first file is stored at a cloud storage location;
determining whether the first set of data references a second set of data included in a second file located at the cloud storage location;
when the first set of data references the second set of data, then:
causing to be deleted at the cloud storage location any references to the second set of data by the first set of data; and
causing to be deleted at the cloud storage location the second file;
determining whether the first set of data is referenced by at least a third set of data included in a third file at the cloud storage location; and
when the first set of data is referenced by at least the third set of data, then:
deleting at the cloud storage location any references to the first set of data by the third set of data; and
storing an indication to delete the first file at the cloud storage location.
9. The computer-implemented method of claim 8 , the computer-implemented method further comprising:
causing the second file to be deleted by converting one or more generic file system commands to one or more vendor-specific calls for the cloud storage location.
10. The computer-implemented method of claim 8 , the computer-implemented method further comprising maintaining a content index for the first, second and third sets of data, and wherein the content index includes file names, logical descriptors for a client computer that originated each file, and a size of each file.
11. The computer-implemented method of claim 8 , wherein the cloud storage location is accessed over an HTTP protocol and wherein the cloud storage location is publicly available via a URL.
12. The computer-implemented method of claim 8 , wherein communications for determining whether the first set of data references the second set of data is performed using one or more application programming interfaces (APIs) provided by the cloud storage location.
13. A computer-implemented method for utilizing cloud storage resources to store at least a first portion of at least one data file within a network attached storage (NAS) device, wherein the NAS device includes a NAS file system and a non-volatile data store, and wherein the NAS device is communicatively coupled to access the cloud storage resources, the computer-implemented method comprising:
accessing calls to or from the NAS file system for reading of data from, or writing of data to, the non-volatile data store of the NAS device,
wherein the at least one data file consists of multiple data blocks,
wherein the non-volatile data store of the NAS device stores the multiple data blocks of the at least one data file;
wherein the NAS file system of the NAS device controls the reading of data from or the writing of data to the multiple data blocks of the at least one data file, and
wherein the accessing includes identifying individual blocks or groups of blocks within the multiple data blocks of the at least one data file that the NAS file system of the NAS device reads data from or writes data to;
based on the accessing, identifying a portion of the multiple data blocks of the at least one data file that satisfies a data storage criteria;
storing content of the at least one data file to a content index; and
automatically transferring the identified portion of the multiple data blocks for storage by the cloud storage resources.
14. The computer-implemented method of claim 13 , wherein the automatically transferring of the identified portion of the multiple data blocks for storage by the cloud storage resources is performed over an HTTP protocol and wherein the cloud storage resources are publicly available via a URL.
15. The computer-implemented method of claim 13 , wherein the automatically transferring of the identified portion of the multiple data blocks for storage by the cloud storage resources is performed using one or more application programming interfaces (APIs) provided by the cloud storage resources.