Storing digital data in storage devices using smart contract and blockchain technology
Embodiments include a method of storing digital data in a blockchain by receiving the digital data from a user, deduplicating the data in a deduplication system of the user to form deduplicated data, and encoding the deduplicated data into an array format for storage on a blockchain. A smart contract is deployed for deduplication on the destination side of the data, and the deduplicated data is encoded into a Binary Aligned Map (BAM) format for storage on the blockchain. A process on the destination side stores the deduplicated array data as a next block in the blockchain only if the next block agrees with the smart contract, and the blockchain is stored in system storage comprising hard disk, solid state, tape, or optical devices.
1. A computer-implemented method of storing digital data on a blockchain comprising:
receiving the digital data from a user;
deduplicating the digital data in a hardware deduplication system to form deduplicated data, wherein the deduplication system executes a data compression process to store only unique data blocks by replacing redundant data blocks with pointers to unique data blocks copies, and further wherein as new data is written to a system, duplicate data chunks are replaced with pointer references to previously stored data;
encoding, in a processor-based encoder, the deduplicated data into an array format for storage on the blockchain;
deploying a smart contract for deduplication of an array comprising the deduplicated data to produce deduplicated array data;
determining that the next block agrees with the smart contract; and
storing the deduplicated array data as a next block in the blockchain.
2. The method of claim 1 wherein the deduplicated array data is formatted in a Binary Aligned Map (BAM) file format.
3. The method of claim 2 wherein the deduplication of the received digital data comprises a similarity-based deduplication process.
4. The method of claim 3 wherein the similarity-based deduplication process selects a nearest base chunk for each sequence in the BAM file using a locality-sensitive hashing (LSH) index and key-value store (KVS) indexing.
5. The method of claim 4 further comprising calculating a hash of the sequence data in the BAM file.
6. The method of claim 5 further comprising:
sending the hash to the LSH;
obtaining an internal LSH key from the hash;
querying a respective LSH hash index; and
joining a list of pointers to candidates in a bigger list.
7. The method of claim 6 wherein the KVS indexing uses unique entries in an optimal similarity search and retrieves values of candidates for deduplication using respective content hashes as keys.
8. The method of claim 7 further comprising calculating an edit distance between each candidate of the candidates using a delta encoding process.
9. The method of claim 8 further comprising:
combining metadata with the data hashed by the LSH to form reduced data;
sending the reduced data to the system storage; and
writing an entry for the reduced data as the next block in the blockchain.
10. The method of claim 1 wherein the system storage comprises at least one of hard disk drive (HDD) storage, solid state device (SSD) storage, tape storage, or optical storage.
11. A method of constructing a unit of data for storage in a blockchain, comprising:
parsing source data formatted in a Binary Aligned Map (BAM) file to create metadata and data;
compressing, in a processor-based compressor, the metadata to produce compressed metadata;
calculating a hash of the data using a locality-sensitive hashing (LSH) index to obtain a list of deduplication candidates of data chunks of the source data for the deduplication system;
sending the list of deduplication candidates representing deduplicated nucleotide sequence data to the Key Value Store (KVS);
deduplicating the data of the deduplication candidates to form the deduplicated nucleotide sequence in a hardware deduplication system by executing a data compression process to store only unique data blocks to replace redundant data blocks with pointers to unique data blocks copies, and wherein as new data is written to a system, duplicate data chunks are replaced with pointer references to previously stored data; and
combining the compressed metadata and the deduplicated nucleotide sequence data to produce reduced data;
deploying, for deduplication, a smart contract comprising the reduced data; and
storing the reduced data in the blockchain.
12. The method of claim 11 further comprising:
deploying the smart contract for destination-side deduplication of the reduced data; and
storing the deduplicated reduced data in system storage;
determining that the next block agrees with the smart contract; and
writing an entry for the reduced data as a next block in the blockchain.
13. The method of claim 12 wherein the system storage comprises at least one of hard disk drive (HDD) storage, solid state device (SSD) storage, tape storage, or optical storage.
14. The method of claim 13 wherein the data comprises previously deduplicated data generated by a deduplication backup system.
15. The method of claim 12 wherein the block size of the next block in on the order of 4 MB or less.