Granted Patent B1

US 12,430,294 · App. 18/806,557 · Granted Sep 30, 2025

Inventors: Prakhar Jain (Sunnyvale, CA); Bart Samwel (Oegstgeest, NL); Burak Yavuz (San Francisco, CA)

Assignee: Databricks, Inc.

G06F16/174

Abstract

Claims (62)

1. A method comprising:

receiving a transaction to modify first data stored in a storage system;

determining that the first data has been compacted with other data during an intervening re-arrangement transaction which yields compacted data;

determining to rollback the intervening re-arrangement transaction; and

in response to determining to rollback the intervening re-arrangement transaction:

removing the first data from the compacted data, and

committing the transaction to modify the first data.

2. The method of claim 1 , where determining to rollback the intervening re-arrangement transaction comprises:

determining a priority level associated with the transaction to modify the first data; and

determining that the priority level associated with the transaction to modify the first data meets or exceeds a threshold priority level.

3. The method of claim 2 , wherein the priority level associated with the transaction to modify the first data is determined based on at least one of: a type of the transaction, a type of the first data being modified by the transaction, a directory in which the first data is stored, and a dataset to which the first data belongs.

4. The method of claim 1 , where determining to rollback the intervening re-arrangement transaction comprises:

determining that a cost associated with re-executing the transaction to modify the first data is greater than a cost associated with rolling back the intervening re-arrangement transaction.

5. The method of claim 1 , where determining to rollback the intervening re-arrangement transaction comprises:

generating, at a client device, a prompt to authorize executing the rollback of the intervening re-arrangement transaction to commit the transaction; and

receiving, from the client device, a response to the prompt that authorizes execution of the rollback of the intervening re-arrangement transaction to commit the transaction.

6. The method of claim 1 , where determining to rollback the intervening re-arrangement transaction comprises:

determining an impact value indicating the impact of rolling back the intervening re-arrangement transaction; and

determining that the impact value indicating the impact of rolling back the intervening re-arrangement transactional is less than a threshold impact value.

7. The method of claim 6 , wherein the impact value indicating the impact of rolling back the intervening re-arrangement transaction is determined based on at least one of: a number of data files that are compacted into the compacted data, a number of rollback operations that have to be performed to rollback the intervening re-arrangement transaction, a size of the first data in respect to a size of the compacted data, and an extent to which the compaction of the first data and the other data has to be performed.

8. The method of claim 1 , wherein removing the first data from the compacted data comprises:

updating an index mapping of data stored by the storage system to indicate that the first data has been moved to a location of the first data before the intervening re-arrangement transaction.

9. A system comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising:

receiving a transaction to modify first data stored in a storage system;

determining that the first data has been compacted with other data during an intervening re-arrangement transaction which yields compacted data;

determining to rollback the intervening re-arrangement transaction; and

in response to determining to rollback the intervening re-arrangement transaction:

removing the first data from the compacted data, and

committing the transaction to modify the first data.

10. The system of claim 9 , where determining to rollback the intervening re-arrangement transaction comprises:

determining a priority level associated with the transaction to modify the first data; and

determining that the priority level associated with the transaction to modify the first data meets or exceeds a threshold priority level.

11. The system of claim 10 , wherein the priority level associated with the transaction to modify the first data is determined based on at least one of: a type of the transaction, a type of the first data being modified by the transaction, a directory in which the first data is stored, and a dataset to which the first data belongs.

12. The system of claim 9 , where determining to rollback the intervening re-arrangement transaction comprises:

determining that a cost associated with re-executing the transaction to modify first data is greater than a cost associated with rolling back the intervening re-arrangement transaction.

13. The system of claim 9 , where determining to rollback the intervening re-arrangement transaction comprises:

generating, at a client device, a prompt to authorize executing the rollback of the intervening re-arrangement transaction to commit the transaction; and

receiving, from the client device, a response to the prompt that authorizes execution of the rollback of the intervening re-arrangement transaction to commit the transaction.

14. The system of claim 9 , where determining to rollback the intervening re-arrangement transaction comprises:

determining an impact value indicating the impact of rolling back the intervening re-arrangement transaction; and

determining that the impact value indicating the impact of rolling back the intervening re-arrangement transactional is less than a threshold impact value.

15. The system of claim 14 , wherein the impact value indicating the impact of rolling back the intervening re-arrangement transaction is determined based on at least one of: a number of data files that are compacted into the compacted data, a number of rollback operations that have to be performed to rollback the intervening re-arrangement transaction, a size of the first data in respect to a size of the compacted data, and an extent to which the compaction of the first data and the other data has to be performed.

16. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing system, cause the computing system to perform operations comprising:

receiving a transaction to modify first data stored in a storage system;

determining that the first data has been compacted with other data during an intervening re-arrangement transaction which yields compacted data;

determining to rollback the intervening re-arrangement transaction; and

in response to determining to rollback the intervening re-arrangement transaction:

removing the first data from the compacted data, and

committing the transaction to modify the first data.

17. The non-transitory computer-readable medium of claim 16 , where determining to rollback the intervening re-arrangement transaction comprises:

determining a priority level associated with the transaction to modify the first data, wherein the priority level associated with the transaction to modify the first data is determined based on at least one of: a type of the transaction, a type of the first data being modified by the transaction, a directory in which the first data is stored, and a dataset to which the first data belongs; and

determining that the priority level associated with the transaction to modify the first data meets or exceeds a threshold priority level.

18. The non-transitory computer-readable medium of claim 16 , where determining to rollback the intervening re-arrangement transaction comprises:

determining that a cost associated with re-executing the transaction to modify the first data is greater than a cost associated with rolling back the intervening re-arrangement transaction.

19. The non-transitory computer-readable medium of claim 16 , where determining to rollback the intervening re-arrangement transaction comprises:

generating, at a client device, a prompt to authorize executing the rollback of the intervening re-arrangement transaction to commit the transaction; and

receiving, from the client device, a response to the prompt that authorizes execution of the rollback of the intervening re-arrangement transaction to commit the transaction.

20. The non-transitory computer-readable medium of claim 16 , where determining to rollback the intervening re-arrangement transaction comprises:

determining an impact value indicating the impact of rolling back the intervening re-arrangement transaction, wherein the impact value indicating the impact of rolling back the intervening re-arrangement transaction is determined based on at least one of: a number of data files that are compacted into the compacted data, a number of rollback operations that have to be performed to rollback the intervening re-arrangement transaction, a size of the first data in respect to a size of the compacted data, and an extent to which the compaction of the first data and the other data has to be performed; and

determining that the impact value indicating the impact of rolling back the intervening re-arrangement transactional is less than a threshold impact value.

Assignments (2)

Continuity (1)

References Cited (9)