Capturing data lake changes
A request may be received to transform records in a data lake that match one or more query criteria. Data lake records that match the criteria may be retrieved. The retrieved records may include two data lake records associated with the same data lake record identifier. A transformed record may be generated based on time stamps associated with the retrieved records.
1. A computer-implemented method comprising:
receiving a request to transform records in a data lake that match one or more query criteria, the data lake configured to store structured, unstructured, and semi-structured data consumed by a plurality of downstream data services in an on-demand computing services environment;
retrieving from the data lake a plurality of data lake records that match the one or more query criteria, the plurality of data lake records including a first data lake record and a second data lake record, the first data lake record being associated with a designated data lake record identifier and a first insertion time identifying when the first data lake record was created, the second data lake record being associated with the same designated data lake record identifier and a second insertion time identifying when the second data lake record was created;
generating a transformed data lake record for at least one of the plurality of downstream data services by transforming the first data lake record to a data format of the downstream data service, the transforming being responsive to determining that the first insertion time precedes the second insertion time; and
transmitting the transformed data lake record to the at least one of the plurality of downstream data services.
2. The computer-implemented method recited in claim 1 , wherein the designated data lake record identifier is associated with a third data lake record corresponding with an update request.
3. The computer-implemented method recited in claim 2 , wherein
the third data lake record is associated with a third insertion time that precedes the second insertion time, and wherein the first insertion time precedes the third insertion time.
4. The computer-implemented method recited in claim 1 , wherein the designated data lake record identifier is associated with a third request to delete the first data lake record, and wherein the third data lake record is associated with a third insertion time.
5. The computer-implemented method recited in claim 4 , wherein the first data lake record is flagged for deletion after a designated period of time has elapsed after the third insertion time.
6. The computer-implemented method recited in claim 1 , wherein a designated one of the data lake partition identifiers is associated with a pointer to a file in the data lake.
7. The computer-implemented method recited in claim 6 , wherein the pointer to the file is a partition key in a Delta Lake change log table.
8. The computer-implemented method recited in claim 1 , wherein the pointer to the file is a URI independent of a file system underlying the data lake.
9. The computer-implemented method recited in claim 1 , wherein the data lake records are stored in one or more third-party cloud computing storage system.
10. The computer-implemented method recited in claim 1 , wherein the data lake is accessible via an on-demand computing services environment providing computing services to a plurality of organizations via the internet.
11. The computer-implemented method recited in claim 10 , wherein the computing services environment includes a multitenant database that stores information associated with the plurality of organizations.
12. A computing system implemented in a cloud computing environment, the computing system configured to perform a method comprising:
receiving a request to transform records in a data lake that match one or more query criteria, the data lake configured to store structured, unstructured, and semi-structured data consumed by a plurality of downstream data services in an on-demand computing services environment;
retrieving from the data lake a plurality of data lake records that match the one or more query criteria, the plurality of data lake records including a first data lake record and a second data lake record, the first data lake record being associated with a designated data lake record identifier and a first insertion time identifying when the first data lake record was created, the second data lake record being associated with the same designated data lake record identifier and a second insertion time identifying when the second data lake record was created;
generating a transformed data lake record for at least one of the plurality of downstream data services by transforming the first data lake record to a data format of the downstream data service, the transforming being responsive to determining that the first insertion time precedes the second insertion time; and
transmitting the transformed data lake record to the at least one of the plurality of downstream data services.
13. The computing system implemented recited in claim 12 , wherein the designated data lake record identifier is associated with a third data lake record corresponding with an update request.
14. The computing system implemented recited in claim 13 , wherein the third data lake record is associated with a third insertion time that precedes the second insertion time, and wherein the first timestamp precedes the third insertion time.
15. The computer system recited in claim 12 , wherein the designated data lake record identifier is associated with a third request to delete the first data lake record, and wherein the third data lake record is associated with a third insertion time, and wherein the first data lake record is flagged for deletion after a designated period of time has elapsed after the third insertion time.
16. One or more non-transitory machine-readable media having instructions stored thereon for performing a method, the method comprising:
receiving a request to transform records in a data lake that match one or more query criteria, the data lake configured to store structured, unstructured, and semi-structured data consumed by a plurality of downstream data services in an on-demand computing services environment;
retrieving from the data lake a plurality of data lake records that match the one or more query criteria, the plurality of data lake records including a first data lake record and a second data lake record, the first data lake record being associated with a designated data lake record identifier and a first insertion time identifying when the first data lake record was created, the second data lake record being associated with the same designated data lake record identifier and a second insertion time identifying when the second data lake record was created;
generating a transformed data lake record for at least one of the plurality of downstream data services by transforming the first data lake record to a data format of the downstream data service, the transforming being responsive to determining that the first insertion time precedes the second insertion time; and
transmitting the transformed data lake record to the at least one of the plurality of downstream data services.
17. The one or more non-transitory machine-readable media recited in claim 16 , wherein the designated data lake record identifier is associated with a third data lake record corresponding with an update request.