Validation modes for DML statements

1 . A system comprising:

at least one hardware processor; and

at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:

receiving a first query, the first query comprising a first set of statements, the first set of statements including at least a first statement for performing a first Data Manipulation Language (DML) operation on a first table;

determining a set of rows that are modified based on performing the first DML operation on the first table, the set of rows including a first set of updated rows;

generating a first delta file based on the determined set of rows, the first delta file comprising a first bitset to indicate a particular set of rows of the first table that have been deleted or updated, and a first set of updated rows;

generating a first updated rows file based on the first set of updated rows;

determining a second set of rows of the first table that have not been modified by performing the first DML operation on the first table;

generating a first residual file based on the second set of rows that have not been modified; and

performing a validation process to validate the first DML operation using at least the first delta file, the first residual file, and a matching Copy on Write (CoW) file.

2 . The system of claim 1 , wherein performing the validation process comprises:

comparing a first particular set of rows generated based on the first delta file and the first residual file with a particular set of rows from the matching CoW file; and

determining whether the first particular set of rows matches the particular set of rows.

3 . The system of claim 1 , wherein performing the validation process comprises:

generating a first hash aggregate for a combined partition resulting from applying a bitset from the first delta file to the first residual file;

generating a second hash aggregate for the matching CoW file; and

comparing the first hash aggregate and the second hash aggregate to validate correctness of the first DML operation.

4 . The system of claim 1 , wherein the operations further comprise:

storing file access information for a validation file in a hidden column of an expression property (EP) file; and

storing file access information for the first residual file in a header field of the EP file.

5 . The system of claim 1 , wherein the operations further comprise:

scheduling deletion of validation files after a configurable time interval; and

identifying leaked validation files using a validation file specific suffix.

6 . The system of claim 1 , wherein performing the validation process comprises:

executing a validation query that performs a comparison using rows from the first delta file and the first residual file against a particular set of rows from the matching CoW file; and

reporting any inconsistencies detected during the comparison.

7 . The system of claim 1 , wherein the operations further comprise:

determining whether to generate validation files based on available computing resources; and

controlling a rate of validation file generation using a probability parameter.

8 . The system of claim 1 , wherein the first DML operation comprises one of:

a delete operation;

an update operation; or

a merge operation.

9 . The system of claim 1 , wherein performing the validation process comprises:

validating change tracking metadata columns when the first table has change tracking enabled.

10 . The system of claim 1 , wherein the operations further comprise:

automatically controlling a validation file generation rate through a validation service;

decreasing a probability of writing new validation files when validation service load exceeds an accepted load;

increasing the probability when the validation service has available capacity.

11 . A method comprising:

receiving a first query, the first query comprising a first set of statements, the first set of statements including at least a first statement for performing a first Data Manipulation Language (DML) operation on a first table;

determining a set of rows that are modified based on performing the first DML operation on the first table, the set of rows including a first set of updated rows;

generating a first delta file based on the determined set of rows, the first delta file comprising a first bitset to indicate a particular set of rows of the first table that have been deleted or updated, and a first set of updated rows;

generating a first updated rows file based on the first set of updated rows;

determining a second set of rows of the first table that have not been modified by performing the first DML operation on the first table;

generating a first residual file based on the second set of rows that have not been modified; and

performing a validation process to validate the first DML operation using at least the first delta file, the first residual file, and a matching Copy on Write (CoW) file.

12 . The method of claim 11 , wherein performing the validation process comprises:

comparing a first particular set of rows generated based on the first delta file and the first residual file with a particular set of rows from the matching CoW file; and

determining whether the first particular set of rows matches the particular set of rows.

13 . The method of claim 11 , wherein performing the validation process comprises:

generating a first hash aggregate for a combined partition resulting from applying a bitset from the first delta file to the first residual file;

generating a second hash aggregate for the matching CoW file; and

comparing the first hash aggregate and the second hash aggregate to validate correctness of the first DML operation.

14 . The method of claim 11 , further comprising:

storing file access information for a validation file in a hidden column of an expression property (EP) file; and

storing file access information for the first residual file in a header field of the EP file.

15 . The method of claim 11 , further comprising:

scheduling deletion of validation files after a configurable time interval; and

identifying leaked validation files using a validation file specific suffix.

16 . The method of claim 11 , wherein performing the validation process comprises:

executing a validation query that performs a comparison using rows from the first delta file and the first residual file against a particular set of rows from the matching CoW file; and

reporting any inconsistencies detected during the comparison.

17 . The method of claim 11 , further comprising:

determining whether to generate validation files based on available computing resources; and

controlling a rate of validation file generation using a probability parameter.

18 . The method of claim 11 , wherein the first DML operation comprises one of:

a delete operation;

an update operation; or

a merge operation.

19 . The method of claim 11 , wherein performing the validation process comprises:

validating change tracking metadata columns when the first table has change tracking enabled.

20 . A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

receiving a first query, the first query comprising a first set of statements, the first set of statements including at least a first statement for performing a first Data Manipulation Language (DML) operation on a first table;

determining a set of rows that are modified based on performing the first DML operation on the first table, the set of rows including a first set of updated rows;

generating a first delta file based on the determined set of rows, the first delta file comprising a first bitset to indicate a particular set of rows of the first table that have been deleted or updated, and a first set of updated rows;

generating a first updated rows file based on the first set of updated rows;

determining a second set of rows of the first table that have not been modified by performing the first DML operation on the first table;

generating a first residual file based on the second set of rows that have not been modified; and

performing a validation process to validate the first DML operation using at least the first delta file, the first residual file, and a matching Copy on Write (CoW) file.