Schema evolution
Techniques for schema mismatch detection and evolution are described. When data is being uploaded into a source table, schema of the data to be uploaded can be compared with the schema for the source table. If a schema mismatch is detected, the schema of the source table can be modified, and the upload can be continued without data loss.
1. A method comprising:
detecting, by one or more execution nodes, a schema mismatch between a file to be ingested and a source table;
generating, by the one or more execution nodes, a message based on detecting the schema mismatch and transmitting the message to a compute service manager;
modifying, by the compute service manager, a schema for the source table based on the message;
quarantining, by the compute service manager, at least one ingest task associated with the file;
compiling, by the compute service manager, a modified query plan for the at least one quarantined ingest task based on the modified schema of the source table; and
executing, by the one or more execution nodes, the quarantined ingest tasks based on the modified query plan to ingest the file into the source table.
2. The method of claim 1 , further comprising:
receiving, by the compute service manager, a notification regarding the file to be ingested into the source table;
generating, by the compute service manager, a query plan for ingesting the file; and
assigning, by the compute service manager, the at least one ingest task based on the query plan to the one or more execution platforms.
3. The method of claim 1 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
4. The method of claim 1 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
5. The method of claim 1 , wherein the message includes information regarding a type of schema mismatch.
6. The method of claim 1 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the message.
7. The method of claim 1 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
8. The method of claim 1 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.
9. A machine-storage medium embodying instructions that, when executed by one or more machines, cause the one or more machines to perform operations comprising:
detecting, by one or more execution nodes, a schema mismatch between a file to be ingested and a source table;
generating, by the one or more execution nodes, a message based on detecting the schema mismatch and transmitting the message to a compute service manager;
modifying, by the compute service manager, a schema for the source table based on the message;
quarantining, by the compute service manager, at least one ingest task associated with the file;
compiling, by the compute service manager, a modified query plan for the at least one quarantined ingest task based on the modified schema of the source table; and
executing, by the one or more execution nodes, the quarantined ingest tasks based on the modified query plan to ingest the file into the source table.
10. The machine-storage medium of claim 9 , further comprising:
receiving, by the compute service manager, a notification regarding the file to be ingested into the source table;
generating, by the compute service manager, a query plan for ingesting the file; and
assigning, by the compute service manager, the at least one ingest task based on the query plan to the one or more execution platforms.
11. The machine-storage medium of claim 9 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
12. The machine-storage medium of claim 9 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
13. The machine-storage medium of claim 9 , wherein the message includes information regarding a type of schema mismatch.
14. The machine-storage medium of claim 9 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the message.
15. The machine-storage medium of claim 9 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
16. The machine-storage medium of claim 9 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.
17. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
detecting, by one or more execution nodes, a schema mismatch between a file to be ingested and a source table;
generating, by the one or more execution nodes, a message based on detecting the schema mismatch and transmitting the message to a compute service manager;
modifying, by the compute service manager, a schema for the source table based on the message;
quarantining, by the compute service manager, at least one ingest task associated with the file;
compiling, by the compute service manager, a modified query plan for the at least one quarantined ingest task based on the modified schema of the source table; and
executing, by the one or more execution nodes, the quarantined ingest tasks based on the modified query plan to ingest the file into the source table.
18. The system of claim 17 , the operations further comprising:
receiving, by the compute service manager, a notification regarding the file to be ingested into the source table;
generating, by the compute service manager, a query plan for ingesting the file; and
assigning, by the compute service manager, the at least one ingest task based on the query plan to the one or more execution platforms.
19. The system of claim 17 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
20. The system of claim 17 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
21. The system of claim 17 , wherein the message includes information regarding a type of schema mismatch.
22. The system of claim 17 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the message.
23. The system of claim 17 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
24. The system of claim 17 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.