Distributed deduplicated storage system
A distributed, deduplicated storage system according to certain embodiments is arranged in a parallel configuration including multiple deduplication nodes. Deduplicated data is distributed across the deduplication nodes. The deduplication nodes can be networked together and communicate with one another according using a light-weight, customized communication scheme (e.g., a scheme based on FTP or HTTP). In some cases, deduplication management information including deduplication signatures and/or other metadata is stored separately from the deduplicated data in deduplication management nodes, improving performance and scalability.
1. A distributed deduplicated storage system comprising one or more hardware processors configured with software instructions, the distributed deduplicated storage system comprising:
a data media agent, wherein the data media agent is configured to:
identify source data from a client device for storage,
split the source data into at least a first and a second data blocks,
transmit in parallel the first data block to a first deduplication database media agent and the second data block to a second deduplication database media agent;
the first deduplication database media agent having first deduplication management information associated therewith,
wherein the first deduplication management information is stored separately from deduplicated data stored in secondary storage devices, the first deduplication database media agent configured to:
determine whether the first data block is stored in a secondary storage device by querying the first deduplication management information; and
the second deduplication database media agent having second deduplication management information associated therewith,
wherein the second deduplication management information is stored separately from the deduplicated data stored in the secondary storage devices, the second deduplication database media agent configured to:
determine whether the second data block is stored in a secondary storage device by querying the second deduplication management information.
2. The distributed deduplicated storage system of claim 1 further comprising the client device, wherein the client device is separate from the data media agent, wherein the client device is configured to transmit data for storage to the data media agent.
3. The distributed deduplicated storage system of claim 2 , wherein the data transmitted to the data media agent is a data block of a file.
4. The distributed deduplicated storage system of claim 2 , wherein the data transmitted to the data media agent includes two or more files.
5. The distributed deduplicated storage system of claim 1 , wherein the first deduplication database media agent generates a first hash signature of the first data block, the second deduplication database media agent generates a second hash signature of the second data block.
6. The distributed deduplicated storage system of claim 1 , wherein the data media agent is a software module executing on the client device.
7. The distributed deduplicated storage system of claim 1 , wherein the client device generates a first hash signature of the first data block and a second hash signature of the second data block.
8. The distributed deduplicated storage system of claim 1 , further including a plurality of data media agents, wherein the plurality of data media agents are in communication with one another and in communication with a plurality of deduplication database media agents.
9. The distributed deduplicated storage system of claim 8 , wherein a first data media agent of the plurality of data media agents provides a path identifier associated with the second data block to a second data media agent in the plurality of data media agents.
10. The distributed deduplicated storage system of claim 8 , wherein a first data media agent of the plurality of data media agents provides block offset information associated with the second data block to a second data media agent in the plurality of data media agents.
11. The distributed deduplicated storage system of claim 7 , wherein the first deduplication database media agent stores a copy of the second hash signature.
12. At least one non-transitory computer-readable medium carrying instructions, which when executed by at least one data processor, executes operations to store data blocks in a distributed deduplicated storage system, the operations comprising:
identifying source data from a client device for storage,
splitting the source data into at least a first and a second data blocks,
transmitting in parallel the first data block to a first deduplication database media agent and the second data block to a second deduplication database media agent,
the first deduplication database media agent having first deduplication management information associated therewith,
wherein the first deduplication management information is stored separately from deduplicated data stored in secondary storage devices;
determining whether the first data block is stored in a secondary storage device by querying the first deduplication management information,
the second deduplication database media agent having second deduplication management information associated therewith,
wherein the second deduplication management information is stored separately from the deduplicated data stored in the secondary storage devices; and
determining whether the second data block is stored in a secondary storage device by querying the second deduplication management information.
13. The non-transitory medium of claim 12 , the operations further comprising transmitting data for backup to a data media agent from the client device, wherein the client device is separate from the data media agent.
14. The non-transitory medium of claim 12 , wherein the first deduplication database media agent generates a first hash signature of the first data block, the second deduplication database media agent generates a second hash signature of the second data block.
15. The non-transitory medium of claim 12 , wherein the client device generates a first hash signature of the first data block and a second hash signature of the second data block.
16. The non-transitory medium of claim 12 , the distributed deduplicated storage system comprises a plurality of data media agents, wherein the plurality of data media agents are in communication with one another and in communication with a plurality of deduplication database media agents.
17. The non-transitory medium of claim 16 , wherein at least one media agent of the plurality of data media agents is a software module executing on the client device.
18. The non-transitory medium of claim 16 , wherein a first data media agent of the plurality of data media agents provides a path identifier associated with the second data block to a second data media agent in the plurality of data media agents.
19. The non-transitory medium of claim 16 , wherein a first data media agent of the plurality of data media agents provides block offset information associated with the second data block to a second data media agent in the plurality of data media agents.
20. The non-transitory medium of claim 15 , wherein the first deduplication database media agent stores a copy of the second hash signature.
21. The non-transitory medium of claim 16 , wherein the plurality of data media agents communicate hash signatures and headers and links between one another without having a shared static mount path configuration.
22. The non-transitory medium of claim 16 , wherein the plurality of data media agents communicate hash signatures and headers without using network shares.