Granted Patent B2

US 12,657,181 · App. 18/313,834 · Granted Jun 16, 2026

Inventors: Nikolaos Romanos Katsipoulakis (Redwood City, CA); Alexander Miller (San Mateo, CA)

Assignee: Snowflake Inc.

G06F16/2379G06F16/22G06F16/2282G06F16/2329G06F16/2343G06F16/2365

Abstract

Claims (80)

1 . A system comprising:

at least one hardware processor; and

a memory storing instructions that cause the at least one hardware processor to perform operations comprising:

determining that a request to commit a transaction has been received;

prior to committing the transaction and after receiving the request to commit the transaction, performing a write operation to store a metadata version of a table associated with the transaction in a metastore;

performing as part of performing the request to commit the transaction, by a compute service manager, a register operation to register a background worker, the background worker being initiated by a fencing manager provided by the compute service manager, the fencing manager being a different component from a transaction manager provided by the compute service manager;

performing, by the background worker, a trigger operation to perform an execution node job on an execution node;

retrieving, by the execution node job, a set of identifiers of active transactions;

performing, by the execution node job, a metadata version validation process comprising:

retrieving a first metadata version, stored in a table persistence object, of the table in the metastore;

retrieving a second metadata version of the table from a datastore;

comparing the first metadata version with the second metadata version;

determining whether the metadata version validation process fails when the first metadata version is less than the second metadata version, and a read timestamp associated with the transaction is not earlier than a write timestamp associated with the second metadata version; and

determining that the metadata version validation process is successful when determining that the metadata version validation process does not fail;

performing, by the execution node job, a write operation to store the metadata version of the table and the set of identifiers in the datastore;

receiving, by the background worker, an indication that the execution node job has successfully completed; and

performing, by the background worker, an unlock operation to unlock the table.

2 . The system of claim 1 , wherein the metastore comprises a metadata database, the datastore comprises a distributed database provided by a storage platform, and a workload analyzer provided by the compute service manager is different from the fencing manager.

3 . The system of claim 1 , wherein prior to performing the execution node job, performing a statement fencing process, the statement fencing process comprising at least performing the write operation to store the metadata version of the table associated with the transaction in the metastore.

4 . The system of claim 3 , wherein the statement fencing process is performed by an execution node worker.

5 . The system of claim 1 ,

wherein determining that the metadata version validation process fails further comprises:

determining that the second metadata version is found in the datastore;

determining that the first metadata version is less than the second metadata version; and

determining that a read timestamp associated with the transaction is not earlier than a write timestamp associated with the second metadata version.

6 . The system of claim 1 , wherein the operations further comprise:

after determining that the metadata version validation process fails, aborting a statement associated with the transaction; and

re-compiling the statement with an updated metadata version.

7 . The system of claim 1 , wherein the operations further comprise:

after retrieving the set of identifiers of active transactions, waiting until each active transaction associated with the set of identifiers has completed.

8 . The system of claim 1 , wherein the operations further comprise:

after receiving the indication that the execution node job has successfully completed, updating a flag in the metastore indicating that a statement fencing process has been successfully performed for the table, the statement fencing process comprising at least performing the write operation to store the metadata version of the table associated with the transaction in the metastore, and updating the flag in the metastore is performed by a driver provided by the compute service manager.

9 . A method comprising:

determining that a request to commit a transaction has been received;

prior to committing the transaction and after receiving the request to commit the transaction, performing a write operation to store a metadata version of a table associated with the transaction in a metastore;

performing as part of performing the request to commit the transaction, by a compute service manager, a register operation to register a background worker, the background worker being initiated by a fencing manager provided by the compute service manager, the fencing manager being a different component from a transaction manager provided by the compute service manager;

performing, by the background worker, a trigger operation to perform an execution node job on an execution node;

retrieving, by the execution node job, a set of identifiers of active transactions;

performing, by the execution node job, a metadata version validation process comprising:

retrieving a first metadata version, stored in a table persistence object, of the table in the metastore;

retrieving a second metadata version of the table from a datastore;

comparing the first metadata version with the second metadata version;

determining whether the metadata version validation process fails when the first metadata version is less than the second metadata version, and a read timestamp associated with the transaction is not earlier than a write timestamp associated with the second metadata version; and

determining that the metadata version validation process is successful when determining that the metadata version validation process does not fail;

performing, by the execution node job, a write operation to store the metadata version of the table and the set of identifiers in a datastore;

receiving, by the background worker, an indication that the execution node job has successfully completed; and

performing, by the background worker, an unlock operation to unlock the table.

10 . The method of claim 9 , wherein the metastore comprises a metadata database, and the datastore comprises a distributed database provided by a storage platform.

11 . The method of claim 9 , wherein prior to performing the execution node job, performing a statement fencing process, the statement fencing process comprising at least performing the write operation to store the metadata version of the table associated with the transaction in the metastore.

12 . The method of claim 11 , wherein the statement fencing process is performed by an execution node worker.

13 . The method of claim 9 ,

wherein determining that the metadata version validation process fails further comprises:

determining that the second metadata version is found in the datastore;

determining that the first metadata version is less than the second metadata version; and

determining that a read timestamp associated with the transaction is not earlier than a write timestamp associated with the second metadata version.

14 . The method of claim 9 , further comprising:

after determining that the metadata version validation process fails, aborting a statement associated with the transaction; and

re-compiling the statement with an updated metadata version.

15 . The method of claim 9 , further comprising:

after retrieving the set of identifiers of active transactions, waiting until each active transaction associated with the set of identifiers has completed.

16 . The method of claim 9 , further comprising:

after receiving the indication that the execution node job has successfully completed, updating a flag in the metastore indicating that a statement fencing process has been successfully performed for the table, the statement fencing process comprising at least performing the write operation to store the metadata version of the table associated with the transaction in the metastore, and updating the flag in the metastore is performed by a driver provided by the compute service manager.

17 . A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

determining that a request to commit a transaction has been received;

prior to committing the transaction and after receiving the request to commit the transaction, performing a write operation to store a metadata version of a table associated with the transaction in a metastore;

performing as part of performing the request to commit the transaction, by a compute service manager, a register operation to register a background worker, the background worker being initiated by a fencing manager provided by the compute service manager, the fencing manager being a different component from a transaction manager provided by the compute service manager;

performing, by the background worker, a trigger operation to perform an execution node job on an execution node;

retrieving, by the execution node job, a set of identifiers of active transactions;

performing, by the execution node job, a metadata version validation process comprising:

retrieving a first metadata version, stored in a table persistence object, of the table in the metastore; and

retrieving a second metadata version of the table from a datastore;

comparing the first metadata version with the second metadata version;

determining whether the metadata version validation process fails when the first metadata version is less than the second metadata version, and a read timestamp associated with the transaction is not earlier than a write timestamp associated with the second metadata version; and

determining that the metadata version validation process is successful when determining that the metadata version validation process does not fail;

performing, by the execution node job, a write operation to store the metadata version of the table and the set of identifiers in a datastore;

receiving, by the background worker, an indication that the execution node job has successfully completed; and

performing, by the background worker, an unlock operation to unlock the table.

18 . The system of claim 1 , wherein performing, by the background worker, the trigger operation to perform the execution node job on the execution node comprises:

determining that the metastore is unavailable; and

in response to determining that the metastore is unavailable, re-executing the execution node job.

Assignments (1)

Continuity (2)

References Cited (31)