Automatic drift detection and handling
In various example embodiments, a system, computer readable medium and method for schema update engine dynamically updating a target data storage system. Incoming data records are received. A front-end schema of the incoming data records is identified. The front-end schema and the current target schema are compared. Based on identifying a difference between the front-end schema and the current target schema, the current target schema is updated in order to be identical to the front-end schema. The current target data file is closed and the incoming data records are stored in a new target data file according to the updated target schema.
1. A computer implemented method comprising:
receiving incoming data records;
identifying a front-end schema of the incoming data records;
accessing a current target schema; and
based on identifying a difference between the front-end schema and the current target schema:
adding a plurality of roll indicators in a plurality of record headers of the incoming data records;
reading the plurality of record headers comprising the plurality of roll indicators;
in response to reading the plurality of record headers comprising the plurality of roll indicators, closing the current target data file; and
storing the incoming data records.
2. The method of claim 1 , wherein identifying the front-end schema of the incoming data records comprises:
collecting the incoming data records into batches;
extracting data fields of the incoming data records of a batch; and
generating a de facto schema based on the extracted data fields, the de facto schema being considered as the front-end schema.
3. The method of claim 2 , wherein generating the de facto schema based on the extracted data fields comprises combining the extracted data fields and removing duplicate extracted data fields.
4. The method of claim 1 , further comprising:
comparing the front-end schema to the current target schema.
5. The method of claim 4 , wherein comparing the front-end schema to the current target schema further comprises:
based on identifying no difference between the front-end schema and the cached target schema, accessing the current target schema, comparing the front-end schema to the current target schema, and updating the cached target schema with the current target schema.
6. The method of claim 4 , wherein comparing the front-end schema to the current target schema further comprises:
comparing the front-end schema to a cached target schema; and
based on identifying a difference between the front-end schema and the cached target schema, accessing the current target schema, comparing the front-end schema to the current target schema, and updating the cached target schema with the current target schema.
7. The method of claim 1 , wherein the difference between the front-end schema and the current target schema is any one or more of an added data field, a deleted data field, a recast data type of data field, or a renamed data field.
8. The method of claim 1 , wherein further comprising:
generating a metadata record describing data operations to be performed on the current target schema to reconcile the identified difference between the front-end schema and the current target schema;
transmitting the metadata record to a target metastore, the target metastore configured to perform the data operations described by the metadata record on the current target schema;
receiving, from the target metastore, an updated target schema that is identical to the front-end schema;
updating the current target schema using the updated target schema received from the target metastore wherein updating the current target schema comprises creating a partition in the current target schema, adding one or more data field to the current target schema, renaming one or more data field of the current target schema, or recasting a type of a data field of the current target schema to a different type of data field.
9. A computer system comprising:
a memory to store computer executable instructions; and
at least one computer processor coupled to the memory to execute the instruction, that upon execution perform operations comprising:
receiving incoming data records;
identifying a front-end schema of the incoming data records;
accessing a current target schema; and
based on identifying a difference between the front-end schema and the current target schema:
adding a plurality of roll indicators in a plurality of record headers of the incoming data records;
reading the plurality of record headers comprising the plurality of roll indicators;
in response to reading the plurality of record headers comprising the plurality of roll indicators, closing the current target data file; and
storing the incoming data records.
10. The system of claim 9 , wherein identifying the front-end schema of the incoming data records comprises:
collecting the incoming data records into batches;
extracting data fields of the incoming data records of a batch; and
generating a de facto schema based on the extracted data fields, the de facto schema being considered as the front-end schema.
11. The system of claim 10 , wherein generating the de facto schema based on the extracted data fields comprises combining the extracted data fields and removing duplicate extracted data fields.
12. The system of claim 9 , further comprising:
comparing the front-end schema to the current target schema.
13. The system of claim 12 , wherein comparing the front-end schema to the current target schema comprises:
comparing the front-end schema to a cached target schema;
based on identifying a difference between the front-end schema and the cached target schema, accessing the current target schema, comparing the front-end schema to the current target schema, and updating the cached target schema with the current target schema.
14. The system of claim 12 wherein comparing the front-end schema to the current target schema further comprises:
based on identifying no difference between the front-end schema and the cached target schema, accessing the current target schema, comparing the front-end schema to the current target schema, and updating the cached target schema with the current target schema.
15. The system of claim 9 , wherein the difference between the front-end schema and the current target schema is an added data field, a deleted data field, a recast data type of data field, or a renamed data field.
16. The system of claim 9 , further comprising:
generating a metadata record describing data operations to be performed on the current target schema to reconcile the identified difference between the front-end schema and the current target schema;
transmitting the metadata record to a target metastore, the target metastore configured to perform the data operations described by the metadata record on the current target schema;
receiving, from the target metastore, an updated target schema that is identical to the front-end schema;
updating the current target schema using the updated target schema received from the target metastore wherein updating the current target schema comprises creating a partition in the current target schema, adding one or more data field to the current target schema, renaming one or more data field of the current target schema, or recasting a type of a data field of the current target schema to a different type of data field.
17. A non-transitory computer readable medium storing instructions thereon, which when executed by a processor cause a computer system to perform operations comprising:
receiving incoming data records;
identifying a front-end schema of the incoming data records;
accessing a current target schema; and
based on identifying a difference between the front-end schema and the current target schema:
adding a plurality of roll indicators in a plurality of record headers of the incoming data records;
reading the plurality of record headers comprising the plurality of roll indicators;
in response to reading the plurality of record headers comprising the plurality of roll indicators, closing the current target data file; and
storing the incoming data records.
18. The computer readable medium of claim 17 , wherein identifying the front-end schema of the incoming data records comprises:
collecting the incoming data records into batches;
extracting data fields of the incoming data records of a batch; and
generating a de facto schema based on the extracted data fields, the de facto schema being considered as the front-end schema.
19. The computer readable medium of claim 18 , wherein generating the de facto schema based on the extracted data fields comprises combining all the extracted data fields and removing duplicate extracted data fields.
20. The computer readable medium of claim 17 , further comprising:
comparing the front-end schema to the current target schema.