Low latency ingestion into a data system
Described herein are techniques for improving transfer of metadata from a metadata database to a database stored in a data system, such as a data warehouse. The metadata may be written into the metadata database with a version stamp, which is monotonic increasing register value, and a partition identifier, which can be generated using attribute values of the metadata. A plurality of readers can scan the metadata database based on version stamp and partition identifier values to export the metadata to a cloud storage location. From the cloud storage location, the exported data can be auto ingested into the database, which includes a journal and snapshot table.
1. A method comprising:
receiving data into a metadata database;
generating a version stamp for the received data;
generating a partition identifier for the received data, the partition identifier being based on one or more attribute values of the received data;
using one or more writer computing resources, writing the received data into the metadata database with the generated version stamp and partition identifier;
using a plurality of reader computing resources, scanning the metadata database and exporting the received data to a storage location based on the version stamp and partition identifier, the plurality of readers being assigned based on partition identifiers; and
changing a number of reader computing resources, the changing of the number of reader computing resources including:
setting a first number of reader computing resources,
detecting a change in usage conditions, and
based on the detected change in usage conditions, setting a second number of reader computing resources, wherein the second number of computing resources is set to go into effect based on a checkpoint relating to version stamp values.
2. The method of claim 1 , wherein the partition identifier is generated using data identification and account identification values.
3. The method of claim 2 , wherein generating the partition identifier includes hashing the data identification and account identification values.
4. The method of claim 1 , further comprising:
changing a number of partitions, including:
setting a first number of partitions;
detecting a change in usage conditions;
based on the detected change in usage conditions, setting a second number partitions to go into effect at a specified time; and
assigning the plurality of reader computing resources based on the second number of partitions to scan the metadata database starting at the specified time.
5. The method of claim 4 , further comprising:
assigning the one or more writer computing resources to write new data to the second number partitions starting at the specified time plus a delay.
6. The method of claim 1 , further comprising:
receiving a notification from a storage indicating that the exported data is stored in the storage; and
in response to the notification, ingesting the exported data from the storage into a database.
7. The method of claim 6 , further comprising:
ingesting the exported data into a journal table in the database;
making the exported data in the journal table accessible for query processing before the exported data is merged with a snapshot table; and
using a background service operating at specified intervals, merging the exported data from the journal table into the snapshot table.
8. The method of claim 1 , wherein the version stamp is a monotonic increasing register value.
9. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving data into a metadata database;
generating a version stamp for the received data;
generating a partition identifier for the received data, the partition identifier being based on one or more attribute values of the received data;
using one or more writer computing resources, writing the received data into the metadata database with the generated version stamp and partition identifier;
using a plurality of reader computing resources, scanning the metadata database and exporting the received data to a storage location based on the version stamp and partition identifier, the plurality of readers being assigned based on partition identifiers; and
changing a number of reader computing resources, the changing of the number of reader computing resources including:
setting a first number of reader computing resources,
detecting a change in usage conditions, and
based on the detected change in usage conditions, setting a second number of reader computing resources, wherein the second number of computing resources is set to go into effect based on a checkpoint relating to version stamp values.
10. The machine-storage medium of claim 9 , wherein the partition identifier is generated using data identification and account identification values.
11. The machine-storage medium of claim 10 , wherein generating the partition identifier includes hashing the data identification and account identification values.
12. The machine-storage medium of claim 9 , further comprising:
changing a number of partitions, including:
setting a first number of partitions;
detecting a change in usage conditions;
based on the detected change in usage conditions, setting a second number partitions to go into effect at a specified time; and
assigning the plurality of reader computing resources based on the second number of partitions to scan the metadata database starting at the specified time.
13. The machine-storage medium of claim 12 , further comprising:
assigning the one or more writer computing resources to write new data to the second number partitions starting at the specified time plus a delay.
14. The machine-storage medium of claim 9 , further comprising:
receiving a notification from a storage indicating that the exported data is stored in the storage; and
in response to the notification, ingesting the exported data from the storage into a database.
15. The machine-storage medium of claim 14 , further comprising:
ingesting the exported data into a journal table in the database;
making the exported data in the journal table accessible for query processing before the exported data is merged with a snapshot table; and
using a background service operating at specified intervals, merging the exported data from the journal table into the snapshot table.
16. The machine-storage medium of claim 9 , wherein the version stamp is a monotonic increasing register value.
17. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving data into a metadata database;
generating a version stamp for the received data;
generating a partition identifier for the received data, the partition identifier being based on one or more attribute values of the received data;
using one or more writer computing resources, writing the received data into the metadata database with the generated version stamp and partition identifier;
using a plurality of reader computing resources, scanning the metadata database and exporting the received data to a storage location based on the version stamp and partition identifier, the plurality of readers being assigned based on partition identifiers; and
changing a number of reader computing resources, the changing of the number of reader computing resources including:
setting a first number of reader computing resources,
detecting a change in usage conditions, and
based on the detected change in usage conditions, setting a second number of reader computing resources, wherein the second number of computing resources is set to go into effect based on a checkpoint relating to version stamp values.
18. The system of claim 17 , wherein the partition identifier is generated using data identification and account identification values.
19. The system of claim 18 , wherein generating the partition identifier includes hashing the data identification and account identification values.
20. The system of claim 17 , the operations further comprising:
changing a number of partitions, including:
setting a first number of partitions;
detecting a change in usage conditions;
based on the detected change in usage conditions, setting a second number partitions to go into effect at a specified time; and
assigning the plurality of reader computing resources based on the second number of partitions to scan the metadata database starting at the specified time.
21. The system of claim 20 , the operations further comprising:
assigning the one or more writer computing resources to write new data to the second number partitions starting at the specified time plus a delay.
22. The system of claim 17 , the operations further comprising:
receiving a notification from the storage indicating that the exported data is stored in the storage; and
in response to the notification, ingesting the exported data from the storage into a database.
23. The system of claim 22 , the operations further comprising:
ingesting the exported data into a journal table in the database;
making the exported data in the journal table accessible for query processing before the exported data is merged with a snapshot table; and
using a background service operating at specified intervals, merging the exported data from the journal table into the snapshot table.
24. The system of claim 17 , wherein the version stamp is a monotonic increasing register value.