In-database workflow orchestration for serverless function
The illustrative embodiments provide a framework for executing data-centric workflows within the database session, thus achieving workflow execution with no additional process creation, virtualization, or network overhead. The user-provided serverless functions that comprise the application workflow are compiled into native-image binary executables and deployed as stored procedures. Using native-image stored procedures greatly reduces memory and execution time overhead compared to traditional serverless computing frameworks. An application workflow is configured using workflow metadata, which specifies the native-image stored procedures that make up the application workflow and transaction boundaries at runtime. The framework ensures low overhead fault tolerance using run-to-completion and exactly-once semantics.
1 . A method comprising:
receiving, at a database system in a database session, a remote query that requires execution of an application workflow based on workflow metadata associated with the application workflow, wherein:
the workflow metadata specifies a plurality of native-image stored procedures to be executed in the application workflow, and
the plurality of native-image stored procedures are stored within the database system;
executing, by a workflow orchestrator, the application workflow to perform functions that require access to one or more database objects in the database system, wherein:
the workflow orchestrator executes within the database session in the database system,
a database session process executes within the database session, and the workflow orchestrator shares the same memory and address space as the database session process, and
executing the application workflow comprises executing the plurality of native-image stored procedures within the database session to access the one or more database objects,
wherein the method is performed by one or more computing devices.
2 . The method of claim 1 , further comprising:
receiving, by a dispatcher executing in a machine connected to the database system, a call to compile a plurality of serverless functions;
performing, by the dispatcher, ahead-of-time compiling of the plurality of serverless functions to form the plurality of native-image stored procedures; and
storing, by the dispatcher, the plurality of native-image stored procedures in the database system.
3 . The method of claim 2 , further comprising:
responsive to the dispatcher receiving the workflow metadata, verifying, by the dispatcher, whether the workflow metadata is valid; and
responsive to determining the workflow metadata is valid, storing the workflow metadata in a workflow metadata table.
4 . The method of claim 2 , wherein the dispatcher is a Representational State Transfer (REST) endpoint.
5 . The method of claim 1 , wherein the workflow metadata specifies transactional boundaries and the workflow orchestrator enforces the specified transactional boundaries.
6 . The method of claim 5 , wherein executing the application workflow comprises:
logging, by the workflow orchestrator, outputs of the plurality of native-image stored procedures in a workflow log; and
determining whether a transaction has executed successfully based on the workflow log.
7 . The method of claim 6 , wherein the workflow orchestrator ensures exactly-once semantics.
8 . The method of claim 1 , wherein the workflow orchestrator commits completed transactions to one or more of the plurality of database objects.
9 . The method of claim 1 , wherein the workflow metadata includes one or more of:
names of the plurality of native-image stored procedures,
data types of input arguments of the plurality of native-image stored procedures,
whether each of the plurality of native-image stored procedures modifies the one or more database objects, or
data flow information of the application workflow.
10 . The method of claim 1 , further comprising:
responsive to completing execution of the application workflow, returning a result to a calling function or requesting user.
11 . The method of claim 1 , wherein executing the plurality of native-image stored procedures comprises loading the plurality of native-image stored procedures into the memory of the database session process.
12 . The method of claim 1 , wherein the plurality of native-image stored procedures access the one or more database objects by calling a backend application programming interface within the database session process.
13 . The method of claim 2 , further comprising creating, by the dispatcher, the one or more database objects required by the application workflow.
14 . One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:
receiving, at a database system in a database session, a remote query that requires execution of an application workflow based on workflow metadata associated with the application workflow,
wherein: the workflow metadata specifies a plurality of native-image stored procedures to be executed in the application workflow, and the plurality of native-image stored procedures are stored within the database system;
executing, by a workflow orchestrator, the application workflow to perform functions that require access to one or more database objects in the database system,
wherein: the workflow orchestrator executes within the database session in the database system,
a database session process executes within the database session, and the workflow orchestrator shares the same memory and address space as the database session process, and
executing the application workflow comprises executing the plurality of native-image stored procedures within the database session to access the one or more database objects.
15 . The one or more non-transitory storage media of claim 14 , wherein the instructions further cause:
receiving, by a dispatcher executing in a machine connected to the database system, a call to compile a plurality of serverless functions;
performing, by the dispatcher, ahead-of-time compiling of the plurality of serverless functions to form the plurality of native-image stored procedures; and
storing, by the dispatcher, the plurality of native-image stored procedures in the database system.
16 . The one or more non-transitory storage media of claim 15 , wherein the instructions further cause:
responsive to the dispatcher receiving the workflow metadata, verifying, by the dispatcher, whether the workflow metadata is valid; and
responsive to determining the workflow metadata is valid, storing the workflow metadata in a workflow metadata table.
17 . The one or more non-transitory storage media of claim 14 , wherein the workflow metadata specifies transactional boundaries and the workflow orchestrator enforces the specified transactional boundaries.
18 . The one or more non-transitory storage media of claim 17 , wherein executing the application workflow comprises:
logging, by the workflow orchestrator, outputs of the plurality of native-image stored procedures in a workflow log; and
determining whether a transaction has executed successfully based on the workflow log.
19 . The one or more non-transitory storage media of claim 14 , wherein the instructions further cause:
responsive to completing execution of the application workflow, returning a result to a calling function or requesting user.