Cross-organization and cross-cloud automated data pipelines
Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.
1. A method comprising:
providing a table in a first account of a multi-tenant network database system;
generating, by a processor, first change data capture information showing changes to the table;
creating a first task in the first account, the first task being defined by a first gate clause;
executing a transaction resulting in a change to the table;
triggering execution of the first task based on commit post-processing of the transaction;
determining that the first gate clause in the first task is satisfied based on the first change data capture information;
in response to determining that the first gate clause is satisfied, executing the first task;
generating a secure view of the table for a second account
generating second change data capture information showing changes to the secure view in the second account
creating a second task in the second account, the second task being defined by a second gate clause;
determining the second gate clause in the second task is satisfied based on the second change data capture information for the secure view; and
in response to determining that the second gate clause is satisfied, executing the second task.
2. The method of claim 1 , further comprising:
maintaining a list of tasks referencing the table.
3. The method of claim 1 , further comprising:
generating a view of the table, wherein the first task directly references the view and wherein the change data capture information directly references the view.
4. The method of claim 1 , further comprising:
triggering execution of the second task based on commit post-processing of the transaction.
5. The method of claim 4 , wherein the first account and second account are in a single deployment of the multi-tenant network database system.
6. The method of claim 4 , wherein the first account is in a first deployment of the multi-tenant network database system and the second account is in second deployment of the multi-tenant network database system.
7. The method of claim 6 , further comprising:
providing a refresh processing element in the second deployment, wherein the secure view references the refresh processing element and wherein the refresh processing element triggers execution of the second task.
8. A non-transitory machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform actions comprising:
providing a table in a first account of a multi-tenant network database system;
generating, by a processor, first change data capture information showing changes to the table;
creating a first task in the first account, the first task being defined by a first gate clause;
executing a transaction resulting in a change to the table;
triggering execution of the first task based on commit post-processing of the transaction;
determining that the first gate clause in the first task is satisfied based on the first change data capture information;
in response to determining that the first gate clause is satisfied, executing the first tasks;
generating a secure view of the table for a second account
generating second change data capture information showing changes to the secure view in the second account
creating a second task in the second account, the second task being defined by a second gate clause;
determining the second gate clause in the second task is satisfied based on the second change data capture information for the secure view; and
in response to determining that the second gate clause is satisfied, executing the second task.
9. The non-transitory machine-storage medium of claim 8 , further comprising:
maintaining a list of tasks referencing the table.
10. The non-transitory machine-storage medium of claim 9 , further comprising:
generating a view of the table, wherein the first task directly references the view and wherein the change data capture information directly references the view.
11. The non-transitory machine-storage medium of claim 9 , further comprising:
triggering execution of the second task based on commit post-processing of the transaction.
12. The non-transitory machine-storage medium of claim 11 , wherein the first account and second account are in a single deployment of the multi-tenant network database system.
13. The non-transitory machine-storage medium of claim 11 , wherein the first account is in a first deployment of the multi-tenant network database system and the second account is in second deployment of the multi-tenant network database system.
14. The non-transitory machine-storage medium of claim 13 , further comprising:
providing a refresh processing element in the second deployment, wherein the secure view references the refresh processing element and wherein the refresh processing element triggers execution of the second task.
15. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform actions comprising:
providing a table in a first account of a multi-tenant network database system;
generating first change data capture information showing changes to the table;
creating a first task in the first account, the first task being defined by a first gate clause;
executing a transaction resulting in a change to the table;
triggering execution of the first task based on commit post-processing of the transaction;
determining that the first gate clause in the first task is satisfied based on the first change data capture information;
in response to determining that the first gate clause is satisfied, executing the first tasks;
generating a secure view of the table for a second account
generating second change data capture information showing changes to the secure view in the second account
creating a second task in the second account, the second task being defined by a second gate clause;
determining the second gate clause in the second task is satisfied based on the second change data capture information for the secure view; and
in response to determining that the second gate clause is satisfied, executing the second task.
16. The system of claim 15 , the operations further comprising:
maintaining a list of tasks referencing the table.
17. The system of claim 15 , the operations further comprising:
generating a view of the table, wherein the first task directly references the view and wherein the change data capture information directly references the view.
18. The system of claim 15 , the operations further comprising:
triggering execution of the second task based on commit post-processing of the transaction.
19. The system of claim 18 , wherein the first account and second account are in a single deployment of the multi-tenant network database system.
20. The system of claim 18 , wherein the first account is in a first deployment of the multi-tenant network database system and the second account is in second deployment of the multi-tenant network database system.
21. The system of claim 20 , the operations further comprising:
providing a refresh processing element in the second deployment, wherein the secure view references the refresh processing element and wherein the refresh processing element triggers execution of the second task.