Systems and methods for a serverless orchestration layer

1 . A method for server-less data management, the method comprising:

receiving a dataset comprising a plurality of partitions;

providing the dataset as an input to a first machine learning model, wherein the first machine learning model is trained based on one of historical data sets or simulated datasets to identify one or more partitions from the plurality of partitions;

receiving, from the first machine learning model, a first output comprising a first partition and a second partition of the plurality of partitions identified based on the dataset;

generating a first kick-off job message associated with the first partition and a second kick-off job message associated with the second partition;

sequencing the first kick-off job message before the second kick-off job message based on a sequencing model;

generating a first job based on the first kick-off job message and a second job based on the second kick-off job message; and

transmitting the first job before the second job based on sequencing the first kick-off job message before the second kick-off job message.

2 . The method of claim 1 , wherein the dataset is received at an orchestration lambda, the orchestration lambda being a part of an orchestration server-less layer.

3 . The method of claim 2 , further comprising initiating, by the orchestration lambda, one or more of a simple queue service (SQS) queue, a job submitter lambda, or an orchestration Elastic Map Reduce (EMR).

4 . The method of claim 1 , further comprising providing the first kick-off job message and the second kick-off job message to a simple queue service (SQS) queue.

5 . The method of claim 1 , wherein the sequencing model is configured to sequence the first kick-off job message before the second kick-off job message based on one or more of a timestamp, an urgency, or an order of importance.

6 . The method of claim 1 , wherein the sequencing model is a machine learning model.

7 . The method of claim 6 , wherein the machine learning model is trained based on one or more of historical kick-off messages, simulated kick-off messages, historical sequences, or simulated sequences.

8 . The method of claim 1 , wherein the sequencing model comprises a second machine learning model and wherein sequencing the first kick-off job message before the second kick-off job message comprises:

providing the first output to the second machine learning model, wherein the second machine learning model is trained based on one or more of historical kick-off messages, simulated kick-off messages, historical sequences, or simulated sequences; and

receiving, from the second machine learning model, a second output comprising the first kick-off job message sequenced before the second kick-off job message.

9 . The method of claim 1 , wherein, to identify the first partition and the second partition, the first machine learning model is configured to detect a threshold difference between a first identifier associated with the first partition and a second identifier associated with the second partition.

10 . A system comprising:

an orchestration lambda;

a simple queue service (SQS) queue;

a job submitter lambda;

an orchestration Elastic Map Reduce (EMR); and

one or more processors configured to:

receive a dataset comprising a plurality of partitions at the orchestration lambda;

generate a dataset kick-off job at the orchestration lambda;

receive the dataset kick-off job at the SQS queue;

receive the dataset kick-off job at the job submitter lambda;

receive the dataset kick-off job at the orchestration EMR;

segregate the dataset into a first partition and a second partition at the orchestration EMR, based on the dataset kick-off job;

generate a first kick-off job message associated with the first partition and a second kick-off job message associated with the second partition, at the orchestration EMR;

receive the first kick-off job message and the second kick-off job message at the SQS queue;

sequence the first kick-off job message before the second kick-off job message based on a sequencing model, by the SQS queue;

receive the first kick-off job message before the second kick-off job message at the job submitter lambda;

generate a first job based on the first kick-off job message and a second job based on the second kick-off job message, at the job submitter lambda; and

transmit, by the job submitter lambda, the first job before the second job based on sequencing the first kick-off job message before the second kick-off job message.

11 . The system of claim 10 , wherein the orchestration lambda is a part of an orchestration server-less layer.

12 . The system of claim 11 , wherein the orchestration lambda is configured to initiate one or more of the SQS queue, the job submitter lambda, or the orchestration EMR.

13 . The system of claim 10 , wherein the segregating the dataset into the first partition and the second partition comprises:

detecting a first identifier associated with the first partition and a second identifier associated with the second partition;

determining that the first identifier is a threshold amount different than the second identifier; and

identifying the first partition and the second partition based on determining that the first identifier is the threshold amount different than the second identifier.

14 . The system of claim 10 , wherein the sequencing model is configured to sequence the first kick-off job message before the second kick-off job message based on one or more of a timestamp, an urgency, or an order of importance.

15 . The system of claim 10 , wherein the sequencing model is a machine learning model.

16 . The system of claim 15 , wherein the machine learning model is trained based on one or more of historical kick-off messages, simulated kick-off messages, historical sequences, or simulated sequences.

17 . The system of claim 10 , wherein segregating the dataset into the first partition and the second partition comprises:

providing the dataset as an input to a first machine learning model, the first machine learning model trained based on one of historical data sets or simulated datasets; and

receiving a first machine learning model output comprising the first partition and the second partition based on the dataset.

18 . The system of claim 17 , wherein the sequencing model comprises a second machine learning model, and wherein sequencing the first kick-off job message before the second kick-off job message comprises:

providing the first machine learning model output to the second machine learning model, the second machine learning model trained based on one or more of historical kick-off messages, simulated kick-off messages, historical sequences, or simulated sequences; and

receiving a second machine learning model output comprising the first kick-off job message sequenced before the second kick-off job message.

19 . A system comprising:

a data storage device storing processor-readable instructions; and

a processor operatively connected to the data storage device and configured to execute the processor-readable instructions to perform operations that include:

receiving a dataset comprising a plurality of partitions;

scanning the dataset to identify a first partition and a second partition, the scanning including:

detecting a first identifier associated with the first partition and a second identifier associated with the second partition;

determining that the first identifier is a threshold amount different than the second identifier; and

identifying the first partition and the second partition based on determining that the first identifier is the threshold amount different than the second identifier;

generating a first kick-off job message associated with the first partition and a second kick-off job message associated with the second partition;

sequencing the first kick-off job message before the second kick-off job message based on a sequencing model;

generating a first job based on the first kick-off job message and a second job based on the second kick-off job message; and

transmitting the first job before the second job based on sequencing the first kick-off job message before the second kick-off job message.

20 . The system of claim 19 , wherein the dataset is provided to an orchestration Elastic Map Reduce (EMR) to perform the scanning.