Configuration driven controls framework

1 . A method comprising:

receiving a first input data for processing at a multi-phase data processing pipeline, wherein the first input data is associated with a first identifier and a first plurality of attributes;

responsive to receiving the first identifier, determining one or more control requirements of a plurality of control requirements associated with a first phase of the multi-phase data processing pipeline, wherein the one or more control requirements is determined at least in part by the first plurality of attributes;

performing, for the first phase of the multi-phase data processing pipeline, operations to generate a set of control requirement results, the operations comprising:

determining a function from a plurality of functions, wherein the function is associated with a respective control requirement of the plurality of control requirements;

executing the function to determine a control requirement result, wherein the control requirement result is associated with a respective attribute of the first plurality of attributes; and

storing the control requirement result;

responsive to generating the set of control requirement results, modifying the first input data based at least in part on the set of control requirement results and in accordance with requirements associated with the first phase;

storing the modified first input data to a data store;

generating a payload comprising the modified first input data, the set of control requirement results, and the first plurality of attributes;

determining, based at least in part on the first phase of the multi-phase data processing pipeline, a second phase; and

transmitting the payload to the second phase of the multi-phase data processing pipeline.

2 . The method of claim 1 , wherein the second phase is an intermediate phase of the multi-phase data processing pipeline.

3 . The method of claim 2 , further comprising:

receiving the payload comprising the modified data, the set of control requirement results, and the first plurality of attributes;

determining, from the set of control requirement results, that one or more tolerance thresholds has been exceeded;

responsive to the determination that one or more tolerance thresholds is exceeded, generating an incident notification;

transmitting the incident notification; and

aborting further processing of the payload.

4 . The method of claim 1 , wherein the second phase is a final phase of the multi-phase data processing pipeline, the final phase comprising:

determining one or more operational requirements associated with a target system, wherein the target system is determined based on at least one of the first identifier or the first plurality of attributes;

validating the modified data against a schema, wherein the schema is associated with the one or more operational requirements associated with the target system; and

for each of the one or more operational requirements, formatting the modified data in accordance with a respective operational requirement.

5 . The method of claim 1 , wherein the payload is generated with a subset of the respective set of control requirements, the subset being determined based on the first plurality of attributes.

6 . The method of claim 1 , further comprising:

receiving second input data, wherein the second input data is associated with a second identifier and a second plurality of attributes;

responsive to receiving the second identifier, determining an absence of control requirements associated with the second input data;

generating, based on the absence of control requirements associated with the second input data, an incident notification; and

transmitting the incident notification.

7 . The method of claim 1 , further comprising:

receiving a second input data, wherein the second input data is associated with a second identifier and a second plurality of attributes;

responsive to receiving the second identifier, determining an absence of control requirements associated with the second input data;

responsive to determining the absence of control requirements associated with the second input data, accessing a set of default control requirements;

modifying the second input data based at least in part on the set of default control requirements and the second plurality of attributes; and

generating a payload comprising the modified second input data, the set of default control requirements, and the second plurality of attributes.

8 . The method of claim 1 , wherein the control requirements comprise at least one of:

a duplicate record check,

a record count verification,

an empty file detection,

a schema validation, or

a checksum verification.

9 . A system comprising:

a processing system; and

a memory configured to store computer-executable instructions; and

a processor configured to access the memory and execute the computer-executable instructions to at least:

receive a first input data for processing at a multi-phase data processing pipeline, wherein the first input data is associated with a first identifier and a first plurality of attributes;

responsive to receiving the first identifier, determining one or more control requirements of a plurality of control requirements associated with a first phase of the multi-phase data processing pipeline, wherein the one or more control requirements is determined at least in part by the first plurality of attributes;

performing, for the first phase of the multi-phase data processing pipeline, operations to generate a set of control requirement results, the operations comprising:

determining a function from a plurality of functions, wherein the function is associated with a respective control requirement of the plurality of control requirement;

executing the function to determine a control requirement result, wherein the control requirement result is associated with a respective attribute of the first plurality of attributes; and

storing the control requirement result;

responsive to generating the set of control requirement results, modifying the first input data based at least in part on the set of control requirement results and in accordance with requirements associated with the first phase;

storing the modified first input data to a data store;

generating a payload comprising the modified first input data, the set of control requirement results, and the first plurality of attributes;

determining, based at least in part on the first phase of the multi-phase data processing pipeline, a second phase; and

transmitting the payload to the second phase of the multi-phase data processing pipeline.

10 . The system of claim 9 , wherein the second phase is an intermediate phase of the multi-phase data processing pipeline.

11 . The system of claim 10 , wherein execution of the computer-executable instructions further causes the processor to at least:

receive the payload comprising the modified data, the set of control requirement results, and the first plurality of attributes;

determine, from the set of control requirement results, that one or more tolerance thresholds has been exceeded;

responsive to the determination that one or more tolerance thresholds is exceeded, generate an incident notification;

transmit the incident notification; and

abort further processing of the payload.

12 . The system of claim 9 , wherein the second phase is a final phase of the multi-phase data processing and wherein execution of the computer-executable instructions further causes the process to at least:

determine one or more operational requirements associated with a target system, wherein the target system is determined based on at least one of the first identifier or the first plurality of attributes;

validate the modified data against a schema, wherein the schema is associated with the one or more operational requirements associated with the target system; and

for each of the one or more operational requirements, format the modified data in accordance with a respective operational requirement.

13 . The system of claim 9 , wherein the payload is generated with a subset of the respective set of control requirements, the subset being determined based on the first plurality of attributes.

14 . The system of claim 9 , wherein execution of the computer-executable instructions further causes the processor to at least:

receive second input data, wherein the second input data is associated with a second identifier and a second plurality of attributes;

responsive to receiving the second identifier, determine an absence of control requirements associated with the second input data;

generate, based on the absence of control requirements associated with the second input data, an incident notification; and

transmit the incident notification.

15 . The system of claim 9 , wherein execution of the computer-executable instructions further causes the processor to at least:

receive a second input data, wherein the second input data is associated with a second identifier and a second plurality of attributes;

responsive to receiving the second identifier, determine an absence of control requirements associated with the second input data; and

responsive to determining the absence of control requirements associated with the second input data, access a set of default control requirements.

16 . A one or more non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors, cause a system to perform operations comprising:

receiving a first input data for processing at a multi-phase data processing pipeline, wherein the first input data is associated with a first identifier and a first plurality of attributes;

responsive to receiving the first identifier, determining one or more control requirements of a plurality of control requirements associated with a first phase of the multi-phase data processing pipeline, wherein the one or more control requirements is determined at least in part by the first plurality of attributes;

performing, for the first phase of the multi-phase data processing pipeline, operations to generate a set of control requirement results, the operations comprising:

determining a function from a plurality of functions, wherein the function is associated with a respective control requirement of the plurality of control requirement;

executing the function to determine a control requirement result, wherein the control requirement result is associated with a respective attribute of the first plurality of attributes; and

storing the control requirement result;

responsive to generating the set of control requirement results, modifying the first input data based at least in part on the set of control requirement results and in accordance with requirements associated with the first phase;

storing the modified first input data to a data store;

generating a payload comprising the modified first input data, the set of control requirement results, and the first plurality of attributes;

determining, based at least in part on the first phase of the multi-phase data processing pipeline, a second phase; and

transmitting the payload to the second phase of the multi-phase data processing pipeline.

17 . The one or more non-transitory computer-readable media of claim 16 , wherein the second phase is an intermediate phase of the multi-phase data processing pipeline.

18 . The one or more non-transitory computer-readable media of claim 17 , the operations further comprising:

receiving the payload comprising the modified data, the set of control requirement results, and the first plurality of attributes;

determining, from the set of control requirement results, that one or more tolerance thresholds has been exceeded;

responsive to the determination that one or more tolerance thresholds is exceeded, generating an incident notification;

transmitting the incident notification; and

aborting further processing of the payload.

19 . The one or more non-transitory computer-readable media of claim 16 , wherein the second phase is a final phase of the multi-phase data processing pipeline, and the operations further comprising:

determining one or more operational requirements associated with a target system, wherein the target system is determined based on at least one of the first identifier or the first plurality of attributes;

validating the modified data against a schema, wherein the schema is associated with the one or more operational requirements associated with the target system; and

for each of the one or more operational requirements, formatting the modified data in accordance with a respective operational requirement.

20 . The one or more non-transitory computer-readable media of claim 16 ,

wherein the payload is generated with a subset of the respective set of control requirements, the subset being determined based on the first plurality of attributes.