Schema evolution
Techniques for schema mismatch detection and evolution are described. When data is being uploaded into a source table, schema of the data to be uploaded can be compared with the schema for the source table. If a schema mismatch is detected, the schema of the source table can be modified, and the upload can be continued without data loss.
1. A method comprising:
receiving, by a compute service manager, a copy command to copy a file into a source table;
generating, by the compute service manager, a query plan to execute the copy command;
assigning, by the compute service manager, one or more execution nodes to execute the query plan;
detecting, by the one or more execution nodes, a schema mismatch between the file and the source table;
generating, by the one or more execution nodes, a message based on detecting the schema mismatch;
modifying, by the compute service manager, a schema for the source table based on the message;
compiling, by the compute service manager, a modified query plan for the copy command based on the modified schema of the source table; and
executing, by the one or more execution nodes, the copy command based on the modified query plan to copy the file into the source table.
2. The method of claim 1 , further comprising:
setting the copy command as a failed command; and
re-trying the failed command, wherein the modified query plan is compiled based on re-trying the failed command.
3. The method of claim 1 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
4. The method of claim 1 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
5. The method of claim 1 , wherein the message includes information regarding a type of schema mismatch.
6. The method of claim 1 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the message.
7. The method of claim 1 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
8. The method of claim 1 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.
9. A machine-storage medium embodying instructions that, when executed by one or more machines, cause the one or more machines to perform operations comprising:
receiving, by a compute service manager, a copy command to copy a file into a source table;
generating, by the compute service manager, a query plan to execute the copy command;
assigning, by the compute service manager, one or more execution nodes to execute the query plan;
detecting, by the one or more execution nodes, a schema mismatch between the file and the source table;
generating, by the one or more execution nodes, a message based on detecting the schema mismatch;
modifying, by the compute service manager, a schema for the source table based on the message;
compiling, by the compute service manager, a modified query plan for the copy command based on the modified schema of the source table; and
executing, by the one or more execution nodes, the copy command based on the modified query plan to copy the file into the source table.
10. The machine-storage medium of claim 9 , further comprising:
setting the copy command as a failed command; and
re-trying the failed command, wherein the modified query plan is compiled based on re-trying the failed command.
11. The machine-storage medium of claim 9 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
12. The machine-storage medium of claim 9 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
13. The machine-storage medium of claim 9 , wherein the message includes information regarding a type of schema mismatch.
14. The machine-storage medium of claim 9 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the message.
15. The machine-storage medium of claim 9 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
16. The machine-storage medium of claim 9 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.
17. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving, by a compute service manager, a copy command to copy a file into a source table;
generating, by the compute service manager, a query plan to execute the copy command;
assigning, by the compute service manager, one or more execution nodes to execute the query plan;
detecting, by the one or more execution nodes, a schema mismatch between the file and the source table;
generating, by the one or more execution nodes, a message based on detecting the schema mismatch;
modifying, by the compute service manager, a schema for the source table based on the message;
compiling, by the compute service manager, a modified query plan for the copy command based on the modified schema of the source table; and
executing, by the one or more execution nodes, the copy command based on the modified query plan to copy the file into the source table.
18. The system of claim 17 , the operations further comprising:
setting the copy command as a failed command; and
re-trying the failed command, wherein the modified query plan is compiled based on re-trying the failed command.
19. The system of claim 17 , wherein the schema mismatch is detected based on the file including one or more columns not in the source table.
20. The system of claim 17 , wherein the schema mismatch is detected based on the file including null values in a designated non-nullable column in the source table.
21. The system of claim 17 , wherein the message includes information regarding a type of schema mismatch.
22. The system of claim 17 , wherein modifying the schema for the source table includes constructing a data definition language statement to add or alter a column based on information in the message.
23. The system of claim 17 , wherein detecting the schema mismatch includes scanning schema information saved in metadata for the file.
24. The system of claim 17 , wherein detecting the schema mismatch includes:
scanning a set of rows from the file, and
determining a schema for the file based on the scanned set of rows.