Verification of Data Provenance for Existing Computer Systems
A proof of provenance system receives database logs from an existing system that does not natively provide data provenance verification. The system produces a local representation of the database (or a portion thereof). The system periodically or in response to another triggering event flattens the local representation of the database to generate an aggregated slice, which is stored in an aggregated slice store. A reduced representation of the aggregated slice (e.g., a hash or checksum) is determined and appended to a distributed ledger, such as a blockchain. The provenance of data from the existing system can then be verified by redetermining the reduced representation and comparing it to the version appended to the distributed ledger.
1 . A method for enabling data provenance verification for an existing system, the method comprising:
receiving, at a proof of provenance system, database logs generated by the existing system, the database logs indicating operations performed on a data set stored by the existing system;
updating a local copy of the data set based on the operations performed on the data set indicated by the database logs, the local copy stored at the proof of provenance system;
aggregating, responsive to a trigger event, information extracted from the local copy of the data set, the aggregating including flattening data objects within the extracted information to generate a hashable aggregated slice;
applying a hash function to the aggregated slice to generate a first hash of the aggregated slice;
committing the first hash to be appended to a distributed ledger in a single transaction;
at a later time, confirming that the aggreged slice has not been modified since generation of the first hash by:
rehashing the aggregated slice to generate a second hash;
querying the distributed ledger using the second hash to confirm the second hash matches the first hash; and
verifying the provenance of the information included in the aggregated slice responsive to the second hash matching the first hash.
2 . The method of claim 1 , further comprising filtering the database logs to generate filtered logs, wherein the local copy is updated based on the filtered logs.
3 . The method of claim 1 , further comprising filtering the local copy to extract the information that is aggregated to generate the aggregated slice.
4 . The method of claim 1 , wherein the trigger event includes at least one of: determining a predefined amount of time has passed since generation of a previous aggregated slice, determining a predefined number of updates have been made to the data set since generation of a previous aggregated slice, or detection of a specific event.
5 . The method of claim 1 , further comprising:
determining that the first hash was not successfully appended to the distributed ledger;
adding the first hash to a buffer that includes one or more additional hashes;
committing the first hash and the one or more additional hashes to be appended to a distributed ledger in a single transaction; and
adding, response to receiving confirmation that the single transaction was successful, metadata to the aggregated slice store indicating an order of the first hash and the one or more additional hashes in the distributed ledger.
6 . The method of claim 1 , wherein the first hash is cryptographically secure and the distributed ledger is a blockchain.
7 . The method of claim 1 , wherein the existing system does not natively provide data provenance verification and the method is performed without making any modification to the existing system.
8 . A proof of provenance system comprising:
a processor; and
a computer-readable media comprising stored instructions that, when executed, cause the processor to:
receive, at a proof of provenance system, database logs generated by the existing system, the database logs indicating operations performed on a data set stored by the existing system;
update a local copy of the data set based on the operations performed on the data set indicated by the database logs, the local copy stored at the proof of provenance system;
aggregate, responsive to a trigger event, information extracted from the local copy of the data set, the aggregating including flattening data objects within the extracted information to generate a hashable aggregated slice;
apply a hash function to the aggregated slice to generate a first hash of the aggregated slice;
commit the first hash to be appended to a distributed ledger in a single transaction;
rehash the aggregated slice to generate a second hash;
query the distributed ledger using the second hash to confirm the second hash matches the first hash;
confirm that the aggregated slice has not been modified since generation of the first hash responsive to the second mash matching the first hash; and
verify the provenance of the information included in the aggregated slice responsive to confirming the aggregated slice has not been modified.
9 . The system of claim 8 , wherein the stored instructions further comprise instructions that, when executed, cause the processor to filter the database logs to generate filtered logs, and wherein the instructions to update the local representation include instructions that, when executed, cause the processor to update the local copy based on the filtered logs.
10 . The system of claim 8 , wherein the stored instructions further comprise instructions that, when executed, cause the processor to filter the local copy to extract the information that is aggregated to generate the aggregated slice.
11 . The system of claim 8 , wherein the trigger event includes at least one of: a predefined amount of time having passed since generation of a previous aggregated slice, a predefined number of updates having been made to the data set since generation of a previous aggregated slice, or a specific event having occurred.
12 . The system of claim 8 , wherein the stored instructions further comprise instructions that, when executed, cause the processor to:
determine that the first hash was not successfully appended to the distributed ledger;
add the first hash to a buffer that includes one or more additional hashes;
commit the first hash and the one or more additional hashes to be appended to a distributed ledger in a single transaction; and
add, response to receiving confirmation that the single transaction was successful, metadata to the aggregated slice store indicating an order of the first hash and the one or more additional hashes in the distributed ledger.
13 . The system of claim 8 , wherein the first hash is cryptographically secure and the distributed ledger is a blockchain.
14 . The system of claim 8 , wherein the existing system does not natively provide data provenance verification and the method is performed without making any modification to the existing system.
15 . A non-transitory computer readable medium configured to store instructions, the instructions when executed by a processor causing the processor to:
receive, at a proof of provenance system, database logs generated by the existing system, the database logs indicating operations performed on a data set stored by the existing system;
update a local copy of the data set based on the operations performed on the data set indicated by the database logs, the local copy stored at the proof of provenance system;
aggregate, responsive to a trigger event, information extracted from the local copy of the data set, the aggregating including flattening data objects within the extracted information to generate a hashable aggregated slice;
apply a hash function to the aggregated slice to generate a first hash of the aggregated slice;
commit the first hash to be appended to a distributed ledger in a single transaction;
rehash the aggregated slice to generate a second hash;
query the distributed ledger using the second hash to confirm the second hash matches the first hash;
confirm that the aggregated slice has not been modified since generation of the first hash responsive to the second mash matching the first hash; and
verify the provenance of the information included in the aggregated slice responsive to confirming the aggregated slice has not been modified.
16 . The non-transitory computer readable medium of claim 15 , wherein the stored instructions further comprise instructions that, when executed, cause the processor to filter the database logs to generate filtered logs, and wherein the instructions to update the local copy include instructions that, when executed, cause the processor to update the local copy based on the filtered logs.
17 . The non-transitory computer readable medium of claim 15 , wherein the stored instructions further comprise instructions that, when executed, cause the processor to filter the local copy of the data to extract the information that is aggregated to generate the aggregated slice.
18 . The non-transitory computer readable medium of claim 15 , wherein the trigger event includes at least one of: a predefined amount of time having passed since generation of a previous aggregated slice, a predefined number of updates having been made to the data set since generation of a previous aggregated slice, or a specific event having occurred.
19 . The non-transitory computer readable medium of claim 15 , wherein the stored instructions further comprise instructions that, when executed, cause the processor to:
determine that the first hash was not successfully appended to the distributed ledger;
add the first hash to a buffer that includes one or more additional hashes;
commit the first hash and the one or more additional hashes to be appended to a distributed ledger in a single transaction; and
add, response to receiving confirmation that the single transaction was successful, metadata to the aggregated slice store indicating an order of the first hash and the one or more additional hashes in the distributed ledger.
20 . The non-transitory computer readable medium of claim 15 , wherein the existing system does not natively provide data provenance verification and the method is performed without making any modification to the existing system.