System for implementing sub-database replication
A method of implementing sub-table replication starts with the processor detecting an update to an entitlements table. The processor performs filtering of a data table based on the update to the entitlements table. The data table including an entitlements column. The processor detects an update to the entitlements column and performs incremental replication of the data table by causing a version-based replication to be executed. Other embodiments are also described herein.
1. A method of implementing sub-table replication comprising:
file-level filtering, by a processor, of a data table based on an update to an entitlements table,
the entitlements table comprising a plurality of entitlements table rows, the entitlements table rows being associated with a plurality of client identifiers, a plurality of consumer account identifiers, and a plurality of provider account identifiers,
the data table comprising a plurality of data table rows associated the client identifiers and including an entitlements column and data content, the entitlements column including the provider account identifiers;
the update to the entitlements table being associated with a first entitlements table row of the entitlement table rows, the update to the entitlements table including a first client identifier of the client identifiers, a first consumer account identifier of the consumer account identifiers, or a first provider account identifier of the provider account identifiers, and
the file-level filtering comprising:
identifying micro-partitions from the plurality of micro-partitions in the data table having metadata associated with the first client identifier, each of the plurality of micro-partitions in the data table comprising one or more of the plurality of data table rows, and
replicating the identified micro-partitions to a first provider account associated with the first provider account identifier;
detecting an update to the entitlements column during the file-level filtering, and
performing a version-based incremental replication of the data table by performing operations comprising:
updating a version identifier associated with the data table,
adding a plurality of micro-partitions in the data table reflecting the update to the entitlements column, and
replicating the data table rows to a plurality of provider accounts associated with the provider account identifiers in the entitlements table using the entitlements column.
2. The method of claim 1 , wherein the entitlements column is populated or updated by a provider user via a client device.
3. The method of claim 1 , wherein the entitlements column comprises a virtual entitlements column.
4. The method of claim 3 , wherein detecting the update to the entitlements column further comprises:
determining that the update to the entitlements table includes a change in one of the provider accounts identifiers in the entitlement table; and
populating the virtual entitlements column based on the change in the one of the provider accounts identifiers in the entitlements table.
5. The method of claim 1 , wherein each of the plurality of micro-partitions in the data table comprises metadata including a minimum client identifier and maximum client identifier.
6. The method of claim 5 , wherein the metadata is associated with the first client identifier when the first client identifier is within a range established by the minimum client identifier and the maximum client identifier.
7. The method of claim 1 , wherein the plurality of provider account identifiers identify provider accounts in a plurality of geographic regions.
8. The method of claim 1 , wherein the entitlements table further comprises a plurality of entitlements table columns storing the consumer account identifiers and the provider account identifiers.
9. The method of claim 1 , further comprising:
receiving a replication request from the first provider account, the replication request including a first consumer account identifier.
10. The method of claim 1 , wherein the update to the entitlement table includes adding a new entitlements table row to the entitlements table, deleting one of the entitlements table rows, or altering information included in one of the entitlements table rows.
11. The method of claim 10 , wherein altering information included in one of the entitlements table rows includes altering the client identifiers, consumer account identifiers, or provider account identifiers.
12. A system implementing sub-table replication comprising:
a processor; and
a memory having instructions stored therein, when executed by the processor, causes the system to perform operations comprising:
file-level filtering, by a processor, of a data table based on an update to an entitlements table,
the entitlements table comprising a plurality of entitlements table rows, the entitlements table rows being associated with a plurality of client identifiers, a plurality of consumer account identifiers, and a plurality of provider account identifiers,
the data table comprising a plurality of data table rows associated the client identifiers and including an entitlements column and data content, the entitlements column including the provider account identifiers;
the update to the entitlements table being associated with a first entitlements table row of the entitlement table rows, the update to the entitlements table including a first client identifier of the client identifiers, a first consumer account identifier of the consumer account identifiers, or a first provider account identifier of the provider account identifiers, and
the file-level filtering comprising:
identifying micro-partitions from the plurality of micro-partitions in the data table having metadata associated with the first client identifier, each of the plurality of micro-partitions in the data table comprising one or more of the plurality of data table rows, and
replicating the identified micro-partitions to a first provider account associated with the first provider account identifier;
detecting an update to the entitlements column during the file-level filtering, and
performing a version-based incremental replication of the data table by performing operations comprising:
updating a version identifier associated with the data table,
adding a plurality of micro-partitions in the data table reflecting the update to the entitlements column, and
replicating the data table rows to a plurality of provider accounts associated with the provider account identifiers in the entitlements table using the entitlements column.
13. The system of claim 12 , wherein the entitlements column is populated or updated by a provider user via a client device.
14. The system of claim 12 , wherein the entitlements column comprises a virtual entitlements column, and wherein detecting the update to the entitlements column further comprises:
determining that the update to the entitlements table includes a change in one of the provider accounts identifiers in the entitlement table; and
populating the virtual entitlements column based on the change in the one of the provider accounts identifiers in the entitlements table.
15. The system of claim 12 , wherein
each of the plurality of micro-partitions in the data table comprises metadata including a minimum client identifier and maximum client identifier.
16. The system of claim 15 , wherein
the metadata is associated with the first client identifier when the first client identifier is within a range established by the minimum client identifier and the maximum client identifier.
17. The system of claim 12 , wherein the update to the entitlement table includes adding a new entitlements table row to the entitlements table, deleting one of the entitlements table rows, or altering information included in one of the entitlements table rows.
18. A non-transitory computer-readable storage medium including instructions that, when processed by a computer, configure the computer to perform operations comprising:
file-level filtering, by a processor, of a data table based on an update to an entitlements table,
the entitlements table comprising a plurality of entitlements table rows, the entitlements table rows being associated with a plurality of client identifiers, a plurality of consumer account identifiers, and a plurality of provider account identifiers,
the data table comprising a plurality of data table rows associated the client identifiers and including an entitlements column and data content, the entitlements column including the provider account identifiers;
the update to the entitlements table being associated with a first entitlements table row of the entitlement table rows, the update to the entitlements table including a first client identifier of the client identifiers, a first consumer account identifier of the consumer account identifiers, or a first provider account identifier of the provider account identifiers, and
the file-level filtering comprising:
identifying micro-partitions from the plurality of micro-partitions in the data table having metadata associated with the first client identifier, each of the plurality of micro-partitions in the data table comprising one or more of the plurality of data table rows, and
replicating the identified micro-partitions to a first provider account associated with the first provider account identifier;
detecting an update to the entitlements column during the file-level filtering, and
performing a version-based incremental replication of the data table by performing operations comprising:
updating a version identifier associated with the data table,
adding a plurality of micro-partitions in the data table reflecting the update to the entitlements column, and
replicating the data table rows to a plurality of provider accounts associated with the provider account identifiers in the entitlements table using the entitlements column.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the entitlements column is populated or updated by a provider user via a client device.
20. The non-transitory computer-readable storage medium of claim 18 , wherein the entitlements column comprises a virtual entitlements column.
21. The non-transitory computer-readable storage medium of claim 20 , wherein detecting the update to the entitlements column further comprises:
determining that the update to the entitlements table includes a change in one of the provider accounts identifiers in the entitlement table; and
populating the virtual entitlements column based on the change in the one of the provider accounts identifiers in the entitlements table.
22. The non-transitory computer-readable storage medium of claim 18 , wherein
each of the plurality of micro-partitions in the data table comprises metadata including a minimum client identifier and maximum client identifier.
23. The non-transitory computer-readable storage medium of claim 22 , wherein
the metadata is associated with the first client identifier when the first client identifier is within a range established by the minimum client identifier and the maximum client identifier.
24. The non-transitory computer-readable storage medium of claim 18 , wherein the update to the entitlement table includes adding a new entitlements table row to the entitlements table, deleting one of the entitlements table rows, or altering information included in one of the entitlements table rows.