Creating local copies of data stored in cloud-based data repositories
Systems and methods for creating local, or on-premises, secondary copies of data managed and/or stored in online repositories are described. In some embodiments, the systems and methods perform a single application programming interface (API) call to a web application level of an online repository to transfer a copy of the data stored in the online repository from the online repository to a local site of a local information management system. Via the local site, the systems and methods access the copy of the data at the site of the local information management system, generate a metadata manifest file that associates metadata files and data files within the copy of the data of the site of the local information management system, and transfer the data files and the metadata manifest file to a media agent for storage to a secondary storage device of the local information management system.
1. A system comprising:
at least one hardware processor; and
computer memory coupled to the at least one hardware processor and storing instructions that configure the system to:
issue at least one representational state transfer-conforming (RESTful) application programming interface (API) call to a cloud-based repository;
based on the at least one RESTful API call, receive at the system, from the cloud-based repository, a copy of current versions of data objects stored in the cloud-based repository;
extract from the copy a current version of and a corresponding metadata file for each of the data objects;
issue one or more client-side object model (CSOM) API calls to the cloud-based repository to obtain at least one older version of at least one of the data objects stored in the cloud-based repository, wherein each older version precedes a corresponding current version of the at least one of the data objects;
based on the one or more CSOM API calls, receive at the system, from the cloud-based repository, one or more older versions of, and corresponding metadata files for, each of the one or more older versions of the at least one of the data objects;
for each data object having a current version and at least one older version, normalize the corresponding metadata files to a format of a metadata file of the current version of the data object;
generate a combined metadata file that combines metadata files corresponding to each of the data objects and includes normalized metadata files; and
store at least some metadata from the combined metadata file at a computing device component of the system, wherein the computing device is distinct from the cloud-based repository.
2. The system of claim 1 , wherein the instructions further configure the system to: store the combined metadata file, the current versions of the data objects, and the one or more older versions at one or more secondary storage devices of the system, wherein the one or more secondary storage devices are distinct from the cloud-based repository.
3. The system of claim 2 , wherein the instructions further configure the system to: maintain an index at the computing device, wherein the index comprises the at least some metadata from the combined metadata file, and wherein the index is configured to track information indicating where at least of the data objects are stored at the one or more secondary storage devices.
4. The system of claim 1 , wherein to generate the combined metadata file, the system is further configured to: generate a metadata manifest file that associates each metadata file to one or more corresponding data object.
5. The system of claim 1 , wherein the cloud-based repository comprises a document management and storage system that includes a database for storing data and further includes a web server that facilitates a client computing device to access the data objects at the cloud-based repository.
6. The system of claim 5 , wherein the combined metadata file includes one or more of: document name information, author information, date of creation information, and date of modification information for each of the data objects in the cloud-based repository.
7. The system of claim 1 , wherein the one or more CSOM API calls to the cloud-based repository obtain one or more data objects of the cloud-based repository at a document hierarchical level of the cloud-based repository.
8. The system of claim 1 , wherein to issue the at least one RESTful API call to the cloud-based repository the system is further configured to perform an export web API call that obtains contents of the cloud-based repository at a web application hierarchical level of the cloud-based repository.
9. The system of claim 1 , wherein the at least one RESTful API call includes a time frame for data objects that were stored in the cloud-based repository within the time frame, and wherein the copy comprises current versions of the data objects that were stored in the cloud-based repository within the time frame.
10. The system of claim 1 , wherein the at least one RESTful API call includes a time frame for data objects in the cloud-based repository that were modified within the time frame, and wherein the copy comprises current versions of the data objects in the cloud-based repository that were modified within the time frame.
11. A computer-implemented method comprising:
by a system comprising at least one hardware processor and computer memory coupled to the at least one hardware processor:
issuing at least one representational state transfer-conforming (RESTful) application programming interface (API) call to a cloud-based repository;
based on the at least one RESTful API call, receiving at the system, from the cloud-based repository, a copy of current versions of data objects stored in the cloud-based repository;
parsing the copy to extract a current version of and a corresponding metadata file for each of the data objects;
issuing one or more client-side object model (CSOM) API calls to the cloud-based repository to obtain at least one older version of at least one of the data objects stored in the cloud-based repository, wherein each older version precedes a corresponding current version of the at least one of the data objects;
based on the one or more CSOM API calls, receiving at the system, from the cloud-based repository, one or more older versions of, and corresponding metadata files for, each of the one or more older versions of the at least one of the data objects;
for each data object having a current version and at least one older version, normalizing the corresponding metadata files to a format of a metadata file of the current version of the data object;
generating a combined metadata file that combines metadata files corresponding to each of the data objects and includes normalized metadata files; and
storing at least some metadata from the combined metadata file at a computing device component of the system.
12. The computer-implemented method of claim 11 , further comprising: storing the combined metadata file, the current versions of the data objects, and the one or more older versions at one or more secondary storage devices of the system, wherein the one or more secondary storage devices are distinct from the cloud-based repository.
13. The computer-implemented method of claim 12 , further comprising: maintaining an index at the computing device, wherein the index comprises the at least some metadata from the combined metadata file, and wherein the index is configured to track information indicating where at least of the data objects are stored at the one or more secondary storage devices.
14. The computer-implemented method of claim 11 , wherein generating the combined metadata file comprises generating a metadata manifest file that associates each metadata file to one or more corresponding data object.
15. The computer-implemented method of claim 11 , wherein the cloud-based repository comprises a document management and storage system that includes a database for storing data and further includes a web server that facilitates a client computing device to access the data objects at the cloud-based repository.
16. The computer-implemented method of claim 15 , wherein the combined metadata file includes one or more of: document name information, author information, date of creation information, and date of modification information for each of the data objects in the cloud-based repository.
17. The computer-implemented method of claim 11 , wherein the one or more CSOM API calls to the cloud-based repository obtain one or more data objects of the cloud-based repository at a document hierarchical level of the cloud-based repository.
18. The computer-implemented method of claim 11 , wherein issuing the at least one RESTful API call to the cloud-based repository comprises performing an export web API call that obtains contents of the cloud-based repository at a web application hierarchical level of the cloud-based repository.
19. The computer-implemented method of claim 11 , wherein the at least one RESTful API call includes a time frame for data objects that were stored in the cloud-based repository within the time frame, and wherein the copy comprises current versions of the data objects that were stored in the cloud-based repository within the time frame.
20. The computer-implemented method of claim 11 , wherein the at least one RESTful API call includes a time frame for data objects in the cloud-based repository that were modified within the time frame, and wherein the copy comprises current versions of the data objects in the cloud-based repository that were modified within the time frame.