Systems and methods for generating a decentralized index for a distributed backup solution

1 . A method for managing a distributed backup system, the method comprising:

generating a plurality of data archives using a plurality of backup agents in the distributed backup system, wherein each respective backup agent is configured to:

create a respective data archive;

build a respective data index while populating the respective data archive such that a data chunk is indexed simultaneously with the data chunk being added to the respective data archive; and

upload the respective data archive and the respective data index to a centralized storage of the distributed backup system;

receiving, by the centralized storage, the plurality of data archives and a plurality of data indexes from the plurality of backup agents; and

merging, by the centralized storage, the plurality of data indexes for access by a user without mounting the plurality of data archives or accessing data chunks within the plurality of data archives.

2 . The method of claim 1 , wherein the centralized storage comprises a plurality of servers each configured to process a data index received from a respective backup agent of the plurality of backup agents.

3 . The method of claim 1 , wherein adding additional backup agents to the plurality of backup agents does not require proportionally scaling an amount of servers in the centralized storage.

4 . The method of claim 1 , wherein each respective backup agent is configured to build the respective data index based on a pre-existing full snapshot comprising a plurality of data chunks.

5 . The method of claim 4 , wherein the respective backup agent is configured to map the data chunk to a specific region of the pre-existing full snapshot when indexing.

6 . The method of claim 4 , wherein the plurality of data chunks are indexed as folders and files.

7 . The method of claim 1 , wherein each respective backup agent of the plurality of backup agents uses a different indexing scheme and includes an index identifier in the respective data index.

8 . The method of claim 7 , wherein the centralized storage is configured to convert the plurality of data indexes into a universal indexing scheme prior to merging.

9 . A system for managing distributed backups, comprising:

at least one memory;

at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:

generate a plurality of data archives using a plurality of backup agents in the distributed backup system, wherein each respective backup agent is configured to:

create a respective data archive;

build a respective data index while populating the respective data archive such that a data chunk is indexed simultaneously with the data chunk being added to the respective data archive; and

upload the respective data archive and the respective data index to a centralized storage of the distributed backup system;

receive, by the centralized storage, the plurality of data archives and a plurality of data indexes from the plurality of backup agents; and

merge, by the centralized storage, the plurality of data indexes for access by a user without mounting the plurality of data archives or accessing data chunks within the plurality of data archives.

10 . The system of claim 9 , wherein the centralized storage comprises a plurality of servers each configured to process a data index received from a respective backup agent of the plurality of backup agents.

11 . The system of claim 9 , wherein adding additional backup agents to the plurality of backup agents does not require proportionally scaling an amount of servers in the centralized storage.

12 . The system of claim 9 , wherein each respective backup agent is configured to build the respective data index based on a pre-existing full snapshot comprising a plurality of data chunks.

13 . The system of claim 12 , wherein the respective backup agent is configured to map the data chunk to a specific region of the pre-existing full snapshot when indexing.

14 . The system of claim 12 , wherein the plurality of data chunks are indexed as folders and files.

15 . The system of claim 9 , wherein each respective backup agent of the plurality of backup agents uses a different indexing scheme and includes an index identifier in the respective data index.

16 . The system of claim 15 , wherein the centralized storage is configured to convert the plurality of data indexes into a universal indexing scheme prior to merging.

17 . A non-transitory computer readable medium storing thereon computer executable instructions for managing a distributed backup system, including instructions for:

generating a plurality of data archives using a plurality of backup agents in the distributed backup system, wherein each respective backup agent is configured to:

create a respective data archive;

build a respective data index while populating the respective data archive such that a data chunk is indexed simultaneously with the data chunk being added to the respective data archive; and

upload the respective data archive and the respective data index to a centralized storage of the distributed backup system;

receiving, by the centralized storage, the plurality of data archives and a plurality of data indexes from the plurality of backup agents; and

merging, by the centralized storage, the plurality of data indexes for access by a user without mounting the plurality of data archives or accessing data chunks within the plurality of data archives.