System and method for distributed node-based data compaction
A system and method for distributed node-based data compaction. The system uses machine learning on data chunks to generate codebooks which compact the data to be stored, processed, or sent with a smaller data profile than uncompacted data. The system uses a data compaction in an existing blockchain fork or implemented in a new blockchain protocol from which nodes that wish to or need to use the blockchain can do so with a reduced storage requirement. The system uses network data compaction across all nodes to increase the speed of and decrease the size of a blockchain's data packets. The system uses data compaction firmware to increase the efficiency at which mining rigs can computationally validate new blocks on the blockchain. The system can be implemented using any combination of the three data compaction services to meet the needs of the desired blockchain technology.
1. A distributed node-based data compaction system, comprising:
a network compaction service, comprising a first plurality of programming instructions stored in a memory and operating on a processor of a computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the processor to:
store a first reference codebook in the memory of each of a plurality of networked computing nodes, the first reference codebook pretrained by machine learning to determine sourceblocks and to associate codewords to each sourceblock;
store a first deconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the first deconstruction algorithm, when operating on a processor of its respective computing node, causes the computing node to:
receive network data for a consensus pool maintained between the plurality of networked computing nodes;
deconstruct the network data for the consensus pool into a plurality of sourceblocks;
encode the first plurality of sourceblocks into a first codeword using the first reference codebook; and
send the first codeword to the consensus pool; and
store a first reconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the first reconstruction algorithm, when operating on a processor of its respective computing node, causes the processor to:
receive the first codeword; and
reconstruct the network data by decoding the first plurality of sourceblocks using the first reference codebook; and
a compaction module, comprising a second plurality of programming instructions stored in a memory and operating on a processor of a computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the processor to:
store a second reference codebook in the memory of each of the plurality of networked computing nodes, the second reference codebook pretrained by machine learning to determine sourceblocks and associated codewords of the consensus pool; and
store a second deconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the first deconstruction algorithm, when operating on a processor of its respective computing node, causes the processor to:
deconstruct a data block into a second plurality of sourceblocks;
encode the second plurality of sourceblocks into a second codeword using the second reference codebook; and
append the second codeword to a compacted blockchain, the compacted blockchain comprising codewords for each data block of the consensus pool.
2. The system of claim 1 , wherein the deconstruction of the network data for the consensus pool generates a sourceblock and association not in the first reference codebook, attach the sourceblock and association to the first codeword, the first reconstruction algorithm updating the first reference codebook.
3. The system of claim 1 , wherein the deconstruction of the data block generates a sourceblock and association not in the second reference codebook, add the sourceblock and association to the second reference codebook.
4. The system of claim 1 , wherein the compaction module further causes the processor to store a second reconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the second reconstruction algorithm, when operating on the processor of its respective computing node, causes the processor to reconstruct the data block by decoding the second plurality of sourceblocks using the second reference codebook.
5. The system of claim 1 , wherein the processor of each of the plurality of networked computing nodes is a multi-core processor, and further comprising a proof system compaction firmware comprising a third plurality of programming instructions stored in the memory and operating on the multi-core processor of each of the plurality of networked computing nodes, wherein the third plurality of programming instructions, when operating on the multi-core processor of its respective computing node, causes the multi-core processor to:
store a third reference codebook embedded as firmware in the memory of each core in the multi-core processor, the third reference codebook pretrained by machine learning to determine sourceblocks and to associate codewords to each sourceblock;
store a third deconstruction algorithm embedded as firmware in the memory of each core in the multi-core processor, wherein the third deconstruction algorithm, when operating on each core, cause the multi-core processor to:
receive data intended for a different core;
deconstruct the data intended for a different core into a third plurality of sourceblocks;
encode the third plurality of sourceblocks into a third codeword using the third reference codebook; and
send the third codeword to the intended core; and
store a third reconstruction algorithm embedded as firmware in the memory of each core in the multi-core processor, wherein the third reconstruction algorithm, when operating on each core, cause the multi-core processor to:
receive the third codeword at the intended core; and
reconstruct the data intended for a different core by decoding the third plurality of sourceblocks using the third reference codebook.
6. The system of claim 5 , wherein the deconstruction of data intended for a different core generates a sourceblock and association not in the third reference codebook, attach the sourceblock and association to the third codeword, the third reconstruction algorithm updating the third reference codebook.
7. The system of claim 1 , wherein each reference codebook is stored as a blockchain.
8. The system of claim 1 , wherein the distributed node-based data compaction system is implemented as a blockchain protocol.
9. The system of claim 1 , wherein the data compaction module permanently deletes a portion from one or more reference codebooks to redact information of the consensus pool.
10. A method for distributed node-based data compaction, comprising the steps of:
storing a first reference codebook in the memory of each of a plurality of networked computing nodes, wherein the first reference codebook is pretrained by machine learning to determine sourceblocks and to associate codewords to each sourceblock;
storing a first deconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the first deconstruction algorithm, when operating on the processor of its respective computing node, causes the processor to:
receive network data for a consensus pool maintained between the plurality of networked computing nodes;
deconstruct the network data for the consensus pool into a plurality of sourceblocks;
encode the first plurality of sourceblocks into a first codeword using the first reference codebook; and
send the first codeword to the consensus pool; and
storing a first reconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the first reconstruction algorithm, when operating on the processor of its respective computing node, causes the processor to:
receive the first codeword; and
reconstruct the network data by decoding the first plurality of sourceblocks using the first reference codebook;
storing a second reference codebook in the memory of each of the plurality of networked computing nodes, the second reference codebook pretrained by machine learning to determine sourceblocks and associated codewords of the consensus pool; and
storing a second deconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the first deconstruction algorithm, when operating on a processor of its respective computing node, causes the processor to:
deconstruct a data block into a second plurality of sourceblocks;
encode the second plurality of sourceblocks into a second codeword using the second reference codebook; and
append the second codeword to a compacted blockchain, the compacted blockchain comprising codewords for each data block of the consensus pool.
11. The method of claim 10 , wherein the deconstruction of the network data for the consensus pool generates a sourceblock and association not in the first reference codebook, attach the sourceblock and association to the first codeword, the first reconstruction algorithm updating the first reference codebook.
12. The method of claim 10 , wherein the deconstruction of the data block generates a sourceblock and association not in the second reference codebook, add the sourceblock and association to the second reference codebook.
13. The method of claim 10 , further comprising the steps of storing a second reconstruction algorithm in the memory of each of the plurality of networked computing nodes, wherein the second reconstruction algorithm, when operating on the processor of its respective computing node, causes the processor to reconstruct the blockchain block by decoding the second plurality of sourceblocks using the second reference codebook.
14. The method of claim 10 , wherein the processor of each of the plurality of networked computing nodes is a multi-core processor, further comprising the steps of:
storing a third reference codebook embedded as firmware in the memory of each core in each of the multi-core processors, the third reference codebook pretrained by machine learning to determine sourceblocks and to associate codewords to each sourceblock;
storing a third deconstruction algorithm embedded as firmware in the memory of each core in each of the multi-core processors, wherein the third deconstruction algorithm, when operating on each core of its respective multi-core processor, causes the multi-core processor to:
receive data intended for a different core;
deconstruct the data intended for a different core into a third plurality of sourceblocks;
encode the third plurality of sourceblocks into a third codeword using the third reference codebook; and
send the third codeword to the intended core; and
storing a third reconstruction algorithm embedded as firmware in the memory of each core in each of the multi-core processors, wherein the third reconstruction algorithm, when operating on each core of its respective multi-core processor, causes the multi-core processor to:
receive the third codeword at the intended core; and
reconstruct the data intended for a different core by decoding the third plurality of sourceblocks using the third reference codebook.
15. The method of claim 14 , wherein the deconstruction of data intended for a different core generates a sourceblock and association not in the third reference codebook, attach the sourceblock and association to the third codeword, the third reconstruction algorithm updating the third reference codebook.
16. The method of claim 10 , wherein each reference codebook is stored as a blockchain.
17. The method of claim 10 , wherein the distributed node-based data compaction method is implemented in a blockchain protocol.
18. The method of claim 10 , further comprising the step of permanently deleting a portion from one or more reference codebooks to redact information of the consensus pool.