Schema evolution
Techniques for schema mismatch detection and evolution are described. When data is being uploaded into a source table, schema of the data to be uploaded can be compared with the schema for the source table. If a schema mismatch is detected, the schema of the source table can be modified, and the upload can be continued without data loss.
1. A method comprising:
receiving, by at least one hardware processor, a notification of a schema mismatch between a file to be ingested and a source table;
modifying a schema for the source table based on the notification;
quarantining at least one ingest task associated with the file to be ingested based on detecting the schema mismatch;
compiling a modified query plan for the at least one quarantined ingest task based on the modified schema of the source table;
executing the at least one quarantined ingest task based on the modified query plan; and
ingesting the file into the source table using the modified schema.
2. The method of claim 1 , further comprising:
receiving a notification regarding the file to be ingested into the source table;
generating, a query plan for ingesting the file; and
assigning the at least one ingest task based on the query plan to one or more execution platforms.
3. The method of claim 1 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
4. The method of claim 1 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
5. The method of claim 1 , wherein the notification includes information regarding a type of schema mismatch.
6. The method of claim 1 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the notification.
7. The method of claim 1 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
8. The method of claim 1 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.
9. A machine-storage medium embodying instructions that, when executed by one or more machines, cause the one or more machines to perform operations comprising:
receiving a notification of a schema mismatch between a file to be ingested and a source table;
modifying a schema for the source table based on the notification;
quarantining at least one ingest task associated with the file to be ingested based on detecting the schema mismatch;
compiling a modified query plan for the at least one quarantined ingest task based on the modified schema of the source table;
executing the at least one quarantined ingest task based on the modified query plan; and
ingesting the file into the source table using the modified schema.
10. The machine-storage medium of claim 9 , further comprising:
receiving a notification regarding the file to be ingested into the source table;
generating, a query plan for ingesting the file; and
assigning the at least one ingest task based on the query plan to one or more execution platforms.
11. The machine-storage medium of claim 9 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
12. The machine-storage medium of claim 9 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
13. The machine-storage medium of claim 9 , wherein the notification includes information regarding a type of schema mismatch.
14. The machine-storage medium of claim 9 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the notification.
15. The machine-storage medium of claim 9 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
16. The machine-storage medium of claim 9 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.
17. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving a notification of a schema mismatch between a file to be ingested and a source table;
modifying a schema for the source table based on the notification;
quarantining at least one ingest task associated with the file to be ingested based on detecting the schema mismatch;
compiling a modified query plan for the at least one quarantined ingest task based on the modified schema of the source table;
executing the at least one quarantined ingest task based on the modified query plan; and
ingesting the file into the source table using the modified schema.
18. The system of claim 17 , the operations further comprising:
receiving a notification regarding the file to be ingested into the source table;
generating, a query plan for ingesting the file; and
assigning the at least one ingest task based on the query plan to one or more execution platforms.
19. The system of claim 17 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
20. The system of claim 17 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
21. The system of claim 17 , wherein the notification includes information regarding a type of schema mismatch.
22. The system of claim 17 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the notification.
23. The system of claim 17 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
24. The system of claim 17 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.