System for list-based database replication
A method of implementing sub-table replication starts with the processor detecting an update to an entitlements table. The processor performs filtering of a data table based on the update to the entitlements table. The data table including an entitlements column. The processor detects an update to the entitlements column and performs incremental replication of the data table by causing a version-based replication to be executed. Other embodiments are also described herein.
1 . A computer-implemented method of implementing sub-table replication comprising:
receiving, by a replication controller of a data platform, a first inclusion list indicating a first subset of rows from a data table of the data platform to replicate to a first provider account of the data platform;
replicating, by the replication controller, the first subset of rows to the first provider account based on the first inclusion list;
responsive to a provider user updating an entitlements table in the first provider account, performing operations comprising:
updating, by the replication controller, a virtual entitlements column stored in the data table indicating a change in provider accounts associated with a client identifier, the virtual entitlements column indicating a provider account associated with the first subset of rows of the data table;
identifying, by the replication controller, a second subset of rows in the data table associated with the client identifier by performing a filtering of the data table using file-level filtering of partitions of the data table based on metadata associated with the client identifier and the virtual entitlements column, the file-level filtering performed using the metadata without opening the partitions;
generating, by the replication controller, a second inclusion list comprising the second subset of rows associated with the client identifier based on the update to the virtual entitlements column; and
performing, by the replication controller, an incremental replication to replicate the second subset of rows to a second provider account based on the second inclusion list.
2 . The computer-implemented method of claim 1 , wherein the first inclusion list and the second inclusion list each further comprise data of a provider account and a region.
3 . The computer-implemented method of claim 1 , further comprising:
receiving the first inclusion list from a first client device associated with a provider user.
4 . The computer-implemented method of claim 3 , further comprising:
receiving the update to the entitlements table in the first provider account from the first client device.
5 . The computer-implemented method of claim 1 , wherein replicating the first subset of rows comprises:
identifying partitions containing the first subset of rows in the data table; and
replicating the identified partitions.
6 . The computer-implemented method of claim 1 , wherein the data table is distributed across a plurality of storage devices.
7 . The computer-implemented method of claim 1 , wherein replicating the second subset of rows comprises transmitting the second subset of rows to the second provider account over a network.
8 . The computer-implemented method of claim 1 , wherein the incremental replication comprises a version-based replication.
9 . A data platform comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the data platform to perform operations comprising:
providing a first provider account;
providing a second provider account; and
providing a replication controller configured to perform first operations comprising:
receiving a first inclusion list indicating a first subset of rows from a data table to replicate to the first provider account;
replicating the first subset of rows to the first provider account based on the first inclusion list;
responsive to a provider user updating an entitlements table in the first provider account, performing second operations comprising:
updating a virtual entitlements column stored in the data table indicating a change in provider accounts associated with a client identifier, the virtual entitlements column indicating a provider account associated with the first subset of rows of the data table;
identifying a second subset of rows in the data table associated with the client identifier by performing a filtering of the data table using file-level filtering of partitions of the data table based on metadata associated with the client identifier and the virtual entitlements column, the file-level filtering performed using the metadata without opening the partitions;
generating a second inclusion list comprising the second subset of rows associated with the client identifier based on the update to the virtual entitlements column; and
performing an incremental replication to replicate the second subset of rows to a second provider account based on the second inclusion list.
10 . The data platform of claim 9 , wherein the first inclusion list and the second inclusion list each further comprise data of a provider account and a region.
11 . The data platform of claim 9 , wherein the operations further comprise:
receiving the first inclusion list from a first client device associated with a provider user.
12 . The data platform of claim 11 , wherein the operations further comprise:
receiving the update to the entitlements table in the first provider account from the first client device.
13 . The data platform of claim 9 , wherein replicating the first subset of rows comprises:
identifying partitions containing the first subset of rows in the data table; and
replicating the identified partitions.
14 . The data platform of claim 9 , wherein the data table is distributed across a plurality of storage devices.
15 . The data platform of claim 9 , wherein replicating the second subset of rows comprises transmitting the second subset of rows to the second provider account over a network.
16 . The data platform of claim 9 , wherein the incremental replication comprises a version-based replication.
17 . A machine-storage medium comprising machine-readable instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by a replication controller of a data platform, a first inclusion list indicating a first subset of rows from a data table of the data platform to replicate to a first provider account of the data platform;
replicating, by the replication controller, the first subset of rows to the first provider account based on the first inclusion list;
responsive to a provider user updating an entitlements table in the first provider account, performing operations comprising:
updating, by the replication controller, a virtual entitlements column stored in the data table indicating a change in provider accounts associated with a client identifier, the virtual entitlements column indicating a provider account associated with the first subset of rows of the data table;
identifying, by the replication controller, a second subset of rows in the data table associated with the client identifier by performing a filtering of the data table using file-level filtering of partitions of the data table based on metadata associated with the client identifier and the virtual entitlements column, the file-level filtering performed using the metadata without opening the partitions;
generating, by the replication controller, a second inclusion list comprising the second subset of rows associated with the client identifier based on the update to the virtual entitlements column; and
performing, by the replication controller, an incremental replication to replicate the second subset of rows to a second provider account based on the second inclusion list.
18 . The machine-storage medium of claim 17 , wherein the first inclusion list and the second inclusion list each further comprise data of a provider account and data of a region.
19 . The machine-storage medium of claim 17 , wherein the operations further comprise:
receiving the first inclusion list from a first client device associated with a provider user.
20 . The machine-storage medium of claim 19 , wherein the operations further comprise:
receiving the update to the entitlements table in the first provider account from the first client device.
21 . The machine-storage medium of claim 17 , wherein replicating the first subset of rows comprises:
identifying partitions containing the first subset of rows in the data table; and
replicating the identified partitions.
22 . The machine-storage medium of claim 17 , wherein the data table is distributed across a plurality of storage devices.
23 . The machine-storage medium of claim 17 , wherein replicating the second subset of rows comprises transmitting the second subset of rows to the second provider account over a network.
24 . The machine-storage medium of claim 17 , wherein the incremental replication comprises a version-based replication.