Database synchronization system in networked zones using blockchain
There are provided systems and methods for a database synchronization system in networked zones using blockchain. An electronic service provider may maintain data stores across different zones that may need to synchronize data across these zones. As such, the subject technology provides for data storage using interplanetary file system (IPFS) technology according to some embodiments with blockchain technology to create a secure and reliable data synchronization system. IPFS may be implemented as a data storage layer and blockchain as a transaction management system, where the IPFS address of data files and synchronization points are stored in a distributed ledger. Integration of the IPFS network with a fabric network can enhance IPFS with the fabric network to create a more secure file sharing platform to improve the transfer of data and database synchronization between different zones.
1. A method relating to securing a data transfer between databases, the method comprising:
accessing, at a first computing device comprising a first data compute engine, information indicating a request has been made to transfer a data item from a source database in a first zone to a target database in a second zone different from the first zone through an encrypted communication channel associated with a distributed ledger network, wherein the first zone comprises a first plurality of computing devices including the first computing device, and wherein the second zone comprises a second plurality of computing devices;
receiving, by the first computing device, the data item from the source database through a networked connection to the source database;
generating, by the first computing device using a hashing algorithm, a hash value based on content of the data item;
causing storage, by the first computing device, of the data item in at least one of the first plurality of computing devices, wherein the first plurality of computing devices are associated with a distributed file storage network corresponding to the distributed ledger network;
causing storage, by the first computing device, of the hash value and additional information associated with the data item into a ledger on at least one of the second plurality of computing devices, wherein the second plurality of computing devices are associated with the distributed ledger network; and
sending, by the first computing device, a request to update a ledger on each of the second plurality of computing devices with state information indicating a first type of transaction that corresponds to storing of the data item in the distributed file storage network, wherein the request
is configured to synchronize the target database with the source database based on the first type of transaction and a second type of transaction.
2. The method of claim 1 , wherein receiving the data item from the source database is performed via a database client of the first computing device.
3. The method of claim 1 , wherein generating the hash value from content of the data item through the hash algorithm is performed via a distributed storage client of the first data compute engine.
4. The method of claim 1 , wherein the first type of transaction comprises a push transaction and the second type of transaction comprises a pull transaction.
5. The method of claim 1 , further comprising:
receiving, using a web application module with an interface to a user device, a configuration file comprising metadata, wherein the metadata comprises one or more of a database identifier identifying the target database, a table identifier identifying a location within a data structure of the target database, a unique identifier indicating an epoch column that provides a synchronization point to the source database, or a channel identifier indicating the encrypted communication channel between the source database and the target database; and storing the configuration file in a configuration table of a data repository communicably coupled to the web application module.
6. The method of claim 1 , wherein the state information, after being received at a second computing device in the second plurality of computing devices, causes the second computing device to acquire the data item from at least one of the first plurality of computing devices associated with the distributed file storage network.
7. The method of claim 1 , wherein the additional information comprises configuration information including one or more of a database identifier identifying the target database, a table identifier identifying a location within a data structure of the target database, a unique identifier indicating an epoch column that provides a synchronization point to the source database, or a channel identifier indicating the encrypted communication channel between the source database and the target database.
8. A first computing device, comprising:
a processor; and
a non-transitory computer-readable storage medium having stored thereon instructions that are executable by the first computing device to cause the first computing device to perform operations comprising:
accessing, at the first computing device, information indicating a request has been made to transfer a data item from a source database in a first zone to a target database in a second zone different from the first zone through an encrypted communication channel associated with a distributed ledger network, wherein the first zone comprises a first plurality of computing devices including the first computing device, and wherein the second zone comprises a second plurality of computing devices;
receiving, by the first computing device, the data item from the source database through a networked connection to the source database;
generating, by the first computing device using a hashing algorithm, a hash value based on content of the data item;
causing storage, by the first computing device, of the data item in at least one of the first plurality of computing devices, wherein the first plurality of computing devices are associated with a distributed file storage network corresponding to the distributed ledger network;
causing storage, by the first computing device, of the hash value and additional information associated with the data item into a ledger on at least one of the second plurality of computing devices, wherein the second plurality of computing devices are associated with the distributed ledger network; and
sending, by the first computing device, a request to update a ledger on each of the second plurality of computing devices with state information indicating a first type of transaction that corresponds to storing of the data item in the distributed file storage network, wherein the request
is configured to synchronize the target database with the source database based on the first type of transaction and a second type of transaction.
9. The first computing device of claim 8 , wherein receiving the data item from the source database is performed via a database client of the first computing device.
10. The first computing device of claim 8 , wherein generating the hash value from content of the data item through the hash algorithm is performed via a distributed storage client of the first data compute engine.
11. The first computing device of claim 8 , wherein the first type of transaction comprises a push transaction and the second type of transaction comprises a pull transaction.
12. The first computing device of claim 8 , wherein the state information, after being received at a second computing device in the second plurality of computing devices, causes the second computing device to acquire the data item from at least one of the first plurality of computing devices associated with the distributed file storage network.
13. The first computing device of claim 8 , wherein the additional information comprises configuration information including one or more of a database identifier identifying the target database, a table identifier identifying a location within a data structure of the target database, a unique identifier indicating an epoch column that provides a synchronization point to the source database, or a channel identifier indicating the encrypted communication channel between the source database and the target database.
14. The first computing device of claim 8 , wherein the operations further comprise:
receiving, using a web application module with an interface to a user device, a configuration file comprising metadata, wherein the metadata comprises one or more of a database identifier identifying the target database, a table identifier identifying a location within a data structure of the target database, a unique identifier indicating an epoch column that provides a synchronization point to the source database, or a channel identifier indicating the encrypted communication channel between the source database and the target database; and storing the configuration file in a configuration table of a data repository communicably coupled to the web application module.
15. A non-transitory computer-readable storage medium having stored thereon instructions that are executable by a first computing device to cause the first computing device to perform operations comprising:
accessing, at the first computing device, information indicating a request has been made to transfer a data item from a source database in a first zone to a target database in a second zone different from the first zone through an encrypted communication channel associated with a distributed ledger network, wherein the first zone comprises a first plurality of computing devices including the first computing device, and wherein the second zone comprises a second plurality of computing devices;
receiving, by the first computing device, the data item from the source database through a networked connection to the source database;
generating, by the first computing device using a hashing algorithm, a hash value based on content of the data item;
causing storage, by the first computing device, of the data item in at least one of the first plurality of computing devices, wherein the first plurality of computing devices are associated with a distributed file storage network corresponding to the distributed ledger network;
causing storage, by the first computing device, of the hash value and additional information associated with the data item into a ledger on at least one of the second plurality of computing devices, wherein the second plurality of computing devices are associated with the distributed ledger network; and
sending, by the first computing device, a request to update a ledger on each of the second plurality of computing devices with state information indicating a first type of transaction that corresponds to storing of the data item in the distributed file storage network, wherein the request
is configured to synchronize the target database with the source database based on the first type of transaction and a second type of transaction.
16. The non-transitory computer-readable storage medium of claim 15 , wherein receiving the data item from the source database is performed via a database client of the first computing device.
17. The non-transitory computer-readable storage medium of claim 15 , wherein generating the hash value from content of the data item through the hash algorithm is performed via a distributed storage client of the first data compute engine.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the first type of transaction comprises a push transaction and the second type of transaction comprises a pull transaction.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the state information, after being received at a second computing device in the second plurality of computing devices, causes the second computing device to acquire the data item from at least one of the first plurality of computing devices associated with the distributed file storage network.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the additional information comprises configuration information including one or more of a database identifier identifying the target database, a table identifier identifying a location within a data structure of the target database, a unique identifier indicating an epoch column that provides a synchronization point to the source database, or a channel identifier indicating the encrypted communication channel between the source database and the target database.