Template-driven multi-tenant workflow processing
Methods, systems, and devices for multi-tenant workflow processing are described. In some cases, a cloud platform may utilize a set of pre-defined batch processes (e.g., workflow templates) and tenant-specific configurations for instantiating and executing tenant-specific batch processes for each tenant of a user. As such, the cloud platform may utilize common data process workflows for each tenant, where a configuration specifies tenant-specific information for the common data process workflows. The workflow templates may include a set of job definitions (e.g., actions for a server to execute) and a schedule defining the frequency for running the templates for a specific project. The configurations may indicate a tenant to execute the workflow templates for, and may include tenant-specific information to override default template information. The cloud platform or a designated server or server cluster may instantiate and execute workflows based on one or more combinations of configurations and indicated workflow templates.
1. A method for managing a plurality of data processing jobs for a plurality of tenants at a multi-tenant database server, comprising:
receiving a set of pre-defined schedules for data processing jobs shared across tenants, wherein a pre-defined schedule for data processing jobs comprises default parameters for one or more compute jobs for the multi-tenant database server to execute;
receiving a plurality of tenant-specific configurations for executing one or more pre-defined schedules for data processing jobs of the set of pre-defined schedules for data processing jobs at runtime, wherein each tenant-specific configuration comprises execution parameters for a respective tenant of the plurality of tenants, and wherein the set of pre-defined schedules for data processing lobs and the plurality of tenant-specific configurations correspond to a first project;
instantiating a plurality of tenant-specific sets of batch processes by applying each configuration of the plurality of tenant-specific configurations to the one or more pre-defined schedules for data processing jobs, wherein a respective tenant-specific set of batch processes of the plurality of tenant-specific sets of batch processes corresponds to a respective tenant and encompasses each data processing job of the one or more pre-defined schedules of data processing jobs for the respective tenant;
executing the plurality of tenant-specific sets of batch processes at the multi-tenant database server based at least in part on the execution parameters for each tenant-specific configuration;
receiving a second set of pre-defined schedules for data processing lobs shared across tenants and a second plurality of tenant-specific configurations corresponding to a second project;
scheduling execution of tenant-specific sets of batch processes for the second project based at least in part on executing the plurality of tenant-specific sets of batch processes for the first project; and
performing machine learning techniques across tenants or across projects.
2. The method of claim 1 , further comprising:
receiving an updated version of a first pre-defined schedule for data processing jobs of the set of pre-defined schedules for data processing jobs shared across tenants; and
instantiating updated tenant-specific sets of batch processes by applying each configuration of the plurality of tenant-specific configurations associated with the first pre-defined schedule for data processing jobs to the updated version of the first pre-defined schedule for data processing jobs.
3. The method of claim 1 , further comprising:
receiving a new tenant-specific configuration indicating pre-defined schedules for data processing jobs of the set of pre-defined schedules for data processing jobs; and
instantiating a number of new tenant-specific sets of batch processes corresponding to a number of indicated pre-defined schedules for data processing jobs by applying the new tenant-specific configuration to each of the indicated pre-defined schedules for data processing jobs.
4. The method of claim 1 , further comprising:
receiving a delete request indicating a tenant of the plurality of tenants;
identifying one or more tenant-specific configurations corresponding to the tenant;
removing the one or more identified tenant-specific configurations from memory; and
removing any tenant-specific sets of batch processes from memory that correspond to a tenant-specific configuration of the identified one or more tenant-specific configurations.
5. The method of claim 1 , further comprising:
receiving a modification request corresponding to a user identifier, wherein the modification request indicates a modification to the set of pre-defined schedules for data processing jobs, the plurality of tenant-specific configurations, or a combination thereof; and
performing an authentication procedure on the user identifier to determine whether to perform the modification.
6. The method of claim 1 , further comprising:
determining server processing resources for executing the one or more compute jobs of each pre-defined schedule for data processing jobs based at least in part on an auto-scaling configuration for the data processing jobs, a queue of the data processing jobs, a dynamic allocation of the server processing resources, a staggering configuration for executing the one or more compute jobs at different times, or a combination thereof.
7. The method of claim 6 , further comprising:
scheduling execution of the plurality of tenant-specific sets of batch processes based at least in part on the determined server processing resources.
8. The method of claim 1 , further comprising:
receiving an execution call from an application programming interface (API), wherein executing the plurality of tenant-specific sets of batch processes is further based at least in part on the execution call.
9. The method of claim 1 , wherein executing the plurality of tenant-specific sets of batch processes comprises:
retrieving, from a database, one or more data records for processing at the multi-tenant database server.
10. The method of claim 9 , further comprising:
implementing a distributed computing cluster to manage data record retrieval or processing.
11. The method of claim 10 , wherein the distributed computing cluster comprises a MapReduce Cluster, a Spark Cluster, or a combination thereof.
12. The method of claim 1 , further comprising:
monitoring execution of the plurality of tenant-specific sets of batch processes for the first project.
13. The method of claim 1 , wherein the execution parameters override one or more of the default parameters for the pre-defined schedule for data processing jobs.
14. The method of claim 1 , further comprising:
storing an execution status and a log of execution-related information at the multi-tenant database server.
15. The method of claim 1 , wherein the set of pre-defined schedules for data processing jobs comprises a set of static directed acyclic graphs (DAGs), wherein a static DAG defines a sequence for the one or more compute jobs of a pre-defined schedule for data processing jobs.
16. The method of claim 1 , wherein each tenant-specific configuration comprises a tenant identifier (ID).
17. An apparatus for managing a plurality of data processing jobs for a plurality of tenants at a multi-tenant database server, comprising:
a processor;
memory in electronic communication with the processor; and
instructions stored in the memory and executable by the processor to cause the apparatus to:
receive a set of pre-defined schedules for data processing jobs shared across tenants, wherein a pre-defined schedule for data processing jobs comprises default parameters for one or more compute jobs for the multi-tenant database server to execute;
receive a plurality of tenant-specific configurations for executing one or more pre-defined schedules for data processing jobs of the set of pre-defined schedules for data processing jobs at runtime, wherein each tenant-specific configuration comprises execution parameters for a respective tenant of the plurality of tenants, and wherein the set of pre-defined schedules for data processing lobs and the plurality of tenant-specific configurations correspond to a first project;
instantiate a plurality of tenant-specific sets of batch processes by applying each configuration of the plurality of tenant-specific configurations to the one or more pre-defined schedules for data processing jobs, wherein a respective tenant-specific set of batch processes of the plurality of tenant-specific sets of batch processes corresponds to a respective tenant and encompasses each data processing job of the one or more pre-defined schedules of data processing jobs for the respective tenant;
execute the plurality of tenant-specific sets of batch processes at the multi-tenant database server based at least in part on the execution parameters for each tenant-specific configuration;
receive a second set of pre-defined schedules for data processing jobs shared across tenants and a second plurality of tenant-specific configurations corresponding to a second project;
schedule execution of tenant-specific sets of batch processes for the second project based at least in part on executing the plurality of tenant-specific sets of batch processes for the first project; and
perform machine learning techniques across tenants or across projects.
18. A non-transitory computer-readable medium storing code for managing a plurality of data processing jobs for a plurality of tenants at a multi-tenant database server, the code comprising instructions executable by a processor to:
receive a set of pre-defined schedules for data processing jobs shared across tenants, wherein a pre-defined schedule for data processing jobs comprises default parameters for one or more compute jobs for the multi-tenant database server to execute;
receive a plurality of tenant-specific configurations for executing one or more pre-defined schedules for data processing jobs of the set of pre-defined schedules for data processing jobs at runtime, wherein each tenant-specific configuration comprises execution parameters for a respective tenant of the plurality of tenants, and wherein the set of pre-defined schedules for data processing lobs and the plurality of tenant-specific configurations correspond to a first project;
instantiate a plurality of tenant-specific sets of batch processes by applying each configuration of the plurality of tenant-specific configurations to the one or more pre-defined schedules for data processing jobs, wherein a respective tenant-specific set of batch processes of the plurality of tenant-specific sets of batch processes corresponds to a respective tenant and encompasses each data processing job of the one or more pre-defined schedules of data processing jobs for the respective tenant;
execute the plurality of tenant-specific sets of batch processes at the multi-tenant database server based at least in part on the execution parameters for each tenant-specific configuration;
receive a second set of pre-defined schedules for data processing lobs shared across tenants and a second plurality of tenant-specific configurations corresponding to a second project;
schedule execution of tenant-specific sets of batch processes for the second project based at least in part on executing the plurality of tenant-specific sets of batch processes for the first project; and
perform machine learning techniques across tenants or across projects.