Management of data events in a datalake environment using data contracts

1 . A computer-implemented method, comprising:

obtaining, from a data producer of a datalake environment, a first web service application programming interface (API) request to create a table, the first web service API request comprising a data contract specifying a first source schema;

provisioning a set of computing resources in a computing resource service provider for processing data events, the set of computing resources comprising:

a message queue;

an intermediate data store; and

a workflow comprising an acquisition phase and an ingestion phase;

obtaining, from the data producer, a second web service API request to publish a data event, the second web service API request comprising:

a network location for data; and

an identifier associated with the table;

storing the data event in the intermediate data store;

provisioning, in response to the second web service API request, the workflow to execute the acquisition phase, wherein execution of the acquisition phase comprises:

extracting, from the data contract, a set of validations applicable to the data event;

determining that the data passes the set of validations, wherein the set of validations comprises a schema validation that determines whether an actual schema of the data complies with an expected schema of the data contract; and

upon determining that the data passes the set of validations, updating the intermediate data store;

provisioning the workflow, in response to completion of the acquisition phase, to execute the ingestion phase, wherein the ingestion phase comprises:

determining a first target schema of a first view;

executing an extract, transform, load (ETL) procedure on the data to map the first source schema to the first target schema to produce first transformed data according to the first view; and

storing the first transformed data;

mapping the data of the data event to one or more views; and

providing the one or more views to a data consumer of the datalake environment.

2 . The computer-implemented method of claim 1 , further comprising:

obtaining, from the data producer, a third web service API request to publish a second data event;

storing the second data event in the intermediate data store;

provisioning the acquisition phase for further:

performing, based on the second data event, a set of validations against the data contract;

determining that the second data event failed to pass the set of validations;

generating a failure event on the message queue;

obtaining, from the data producer, an update to the second data event or the data contract;

performing the set of validations based on the update; and

updating the intermediate data store in response to successfully validating that the second data event based on the update.

3 . The computer-implemented method of claim 2 , wherein the update comprises a second source schema that is backwards compatible with the first source schema.

4 . The computer-implemented method of claim 1 , wherein the intermediate data store is a key value store.

5 . A system, comprising:

one or more processors; and

memory storing executable instructions that, as a result of execution by the one or more processors, cause the system to:

obtain, from a data producer, a first request to publish a first data event, the first request specifying a table;

store the first data event in a data store;

provision a workflow, in response to the first request, to execute a first phase comprising an acquisition phase and a second phase comprising an ingestion phase, the first phase of the workflow comprising steps to:

obtain a first data contract for the table, the first data contract specifying at least a first source schema;

determine, from a first set of validations encoded in the first data contract, a first set of validations that includes a first schema validation according to the first source schema, wherein the first schema validation indicates an expected schema format for data events generated by the data producer;

perform the first set of validations on the first data event to determine whether the first data event is valid, wherein the first set of validations comprises a validation that an actual schema format of the first data event with the expected schema format; and

write, to the data store, an indication of whether the first data event is valid;

provision the workflow to execute the second phase, the second phase of the workflow comprising steps to:

obtain the first data event from the data store, contingent upon the first data event being valid;

determine a first target schema associated with a first view of a data consumer of the first data event; and

execute an extract, transform, load (ETL) procedure on data of the first data event to produce first transformed data according to the first target schema; and

make the first transformed data available to a data consumer.

6 . The system of claim 5 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:

obtain, from the data producer, a second request to publish a second data event, the second request specifying the table;

perform the first set of validations on the second data event to determine whether the second data event is valid; and

based on the second data event being invalid, generate a failure event in a message queue for the data producer.

7 . The system of claim 6 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:

obtain, from the data producer, a second data contract specifying a second source schema;

persisting the second data contract in a metadata store;

determine, from a second set of validations different from the first set of validations that is encoded in the second data contract, a second schema validation according to the second source schema;

perform the second set of validations on the second data event to determine whether the second data event is valid;

based on the second data event being valid, process second data of the second data event to produce second transformed data according to the first target schema; and

make the second transformed data available to the data consumer.

8 . The system of claim 7 , wherein the second data contract is backwards compatible with the first data contract.

9 . The system of claim 6 , wherein the first data event and the second data event share a same publish time and are sequenced to indicate an order in which the first data event and the second data event are to be processed.

10 . The system of claim 5 , wherein:

the first request comprises a data payload; and

the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to store the data payload in a temporary data store.

11 . The system of claim 5 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:

determine that the first data contract comprises a query statement; and

remap the data of the first data event to modified data based on the query statement.

12 . The system of claim 5 , wherein the system comprises a plurality of machines of a computing resource service provider that collectively comprise the memory that stores the executable instructions.

13 . A computer-implemented method, comprising:

obtaining, from a data producer, a first request to publish a first data event, the first request specifying a table;

store the first data event in a data store;

provision a workflow, in response to the first request, to execute a first phase comprising an acquisition phase and a second phase comprising an ingestion phase, wherein the first phase of the workflow comprises:

obtaining a data contract for the table, the data contract specifying at least a first source schema;

determining, from a first set of validations encoded in the data contract, a first set of validations that includes a first schema validation according to the first source schema, wherein the first schema validation indicates an expected schema format for data events generated by the data producer;

performing the first set of validations on the first data event to determine whether the first data event is valid, comprising validating an actual schema format of the first data event against the expected schema format; and

upon determining that the first data event is valid, writing, to the data store, an indication of whether the first data event is valid;

provisioning the workflow to execute the second phase, wherein the second phase of the workflow comprises:

obtaining the first data event from the data store, contingent upon the first data event being valid;

determining a first target schema associated with a first view of a data consumer of the first data event; and

executing an extract, transform, load (ETL) procedure on the data of the first data event to produce transformed data according to the first target schema; and

making the transformed data available to a data consumer.

14 . The method of claim 13 , further comprising:

obtaining, from the data producer, a second request to publish a second data event, the second request specifying the table;

performing the first set of validations on the second data event to determine whether the second data event is valid; and

based on the second data event being invalid, generating a failure event in a message queue that is accessible by the data producer.

15 . The method of claim 14 , further comprising:

obtaining, from the data producer, an updated second data event;

performing the first set of validations on the updated second data event to determine whether the updated second data event is valid;

based on the updated second data event being valid, processing updated second data of the updated second data event to produce second transformed data according to the first target schema; and

making the second transformed data available to the data consumer.

16 . The method of claim 14 , wherein the first data event and the second data event share a same publish time and are sequenced to indicate an order in which the first data event and the second data event are to be processed.

17 . The method of claim 13 , wherein the first request comprises a data object location where the data is located.

18 . The method of claim 13 , further comprising:

mapping the transformed data to a view based on a one-to-one mapping defined in the data contract.

19 . The method of claim 13 , wherein the first set of validations comprises a file format validation of the data.

20 . The method of claim 13 , wherein the first phase of the workflow is executed by a first set of machines of a computing resource service provider and the second phase of the workflow is executed by a second set of machines of the computing resource service provider.