Hyper-scale P2P deduplicated storage system using a distributed ledger
One example method includes receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node, performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry, making a determination that a replication factor X has not been met, and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.
1. A method, comprising:
receiving from a node, in a hyper-scale, peer-to-peer, de-duplicated storage system (HSAN) that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node;
performing a challenge-and-response process with the node to verify that the node has a copy of the data segment that was the subject of the entry;
making a determination that a replication factor X has not been met, wherein the replication factor X specifies a maximum number X of copies of the data segment that will be stored in the HSAN; and
adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.
2. The method as recited in claim 1 , further comprising splitting backup data into a plurality of data segments that includes the data segment, and hashing the data segment.
3. The method as recited in claim 1 , wherein the distributed ledger does not permit an HSAN node access to the data segment upon which the entry is based unless that HSAN node has demonstrated possession of a copy of that data segment.
4. The method as recited in claim 1 , wherein the distributed ledger is a public distributed ledger that permits any HSAN node to access the entry.
5. The method as recited in claim 1 , wherein the distributed ledger is a public distributed ledger that includes one or more entries that point to public data accessible by all nodes of the HSAN.
6. The method as recited in claim 1 , further comprising adding a node to the HSAN upon mutual agreement of other nodes of the HSAN.
7. The method as recited in claim 1 , wherein metadata concerning the data segment identified in the request is omitted from the request.
8. The method as recited in claim 1 , wherein use of the replication factor X ensures that no more than X copies of a data segment will be stored in the HSAN.
9. The method as recited in claim 1 , wherein entries are stored in the distributed ledger as a blockchain.
10. The method as recited in claim 1 , further comprising performing any one or more of the HSAN operations: IS_PRESENT; REF_DATA; GET_DATA; DEREF_DATA; DEL_DATA; CHECK_DATA; and/or REPL_DATA.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving from a node, in a hyper-scale, peer-to-peer, de-duplicated storage system (HSAN) that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node;
performing a challenge-and-response process with the node to verify that the node has a copy of the data segment that was the subject of the entry;
making a determination that a replication factor X has not been met, wherein the replication factor X specifies a maximum number X of copies of the data segment that will be stored in the HSAN; and
adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.
12. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise splitting backup data into a plurality of data segments that includes the data segment, and hashing the data segment.
13. The non-transitory storage medium as recited in claim 11 , wherein the distributed ledger does not permit an HSAN node access to the data segment upon which the entry is based unless that HSAN node has demonstrated possession of a copy of that data segment.
14. The non-transitory storage medium as recited in claim 11 , wherein the distributed ledger is a public distributed ledger that permits any HSAN node to access the entry.
15. The non-transitory storage medium as recited in claim 11 , wherein the distributed ledger is a public distributed ledger that includes one or more entries that point to public data accessible by all nodes of the HSAN.
16. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise adding a node to the HSAN upon mutual agreement of other nodes of the HSAN.
17. The non-transitory storage medium as recited in claim 11 , wherein metadata concerning the data segment identified in the request is omitted from the request.
18. The non-transitory storage medium as recited in claim 11 , wherein use of the replication factor X ensures that no more than X copies of a data segment will be stored in the HSAN.
19. The non-transitory storage medium as recited in claim 11 , wherein entries are stored in the distributed ledger as a blockchain.
20. The non-transitory storage medium as recited in claim 11 , further comprising performing any one or more of the HSAN operations: IS_PRESENT; REF_DATA; GET_DATA; DEREF_DATA; DEL_DATA; CHECK_DATA; and/or REPL_DATA.