Efficient locking of large data collections
View Patent ↗The present disclosure provides systems and techniques for efficient locking of datasets in a database when updates to a dataset may be delayed. A method may include accumulating a plurality of updates to a first set of one or more values associated with one or more features. The first set of one or more values may be stored within a first database column. Next, it may be determined that a first database column update aggregation rule is satisfied. A lock assigned to at least a portion of at least a first database column may be acquired. Accordingly, one or more values in the first set within the first database column may be updated based on the plurality of updates. In an implementation, the first set of one or more values may be associated with the first lock.
1. A method comprising:
storing a database that includes a plurality of database columns that each include a plurality of entries each having a respective value, wherein the plurality of database columns includes a first column;
associating a first lock with a first lock portion of the database;
associating a second lock with a second lock portion of the database, wherein the second lock portion does not overlap the first lock portion of the database and wherein the first lock portion of the database is more frequently updated than the second lock portion of the database;
accumulating a plurality of first updates, each first update being an update to a value in the first lock portion of the database, prior to acquiring the first lock, until a first predetermined update aggregation rule is satisfied;
in response to determining that the first predetermined update aggregation rule is satisfied, acquiring the first lock associated with the first lock portion of the database and then updating the database by applying the accumulated first updates to the first lock portion of the database;
accumulating a plurality of second updates, each second update being an update to a value in the second lock portion of the database, prior to acquiring the second lock, until a second predetermined update aggregation rule is satisfied;
in response to determining that the second predetermined update aggregation rule is satisfied, acquiring the second lock associated with the second lock portion of the database and then updating the database by applying the accumulated second updates to the second lock portion of the database; and
associating one or more third locks with respective third lock portions of the database, wherein the third lock portions do not overlap each other, the first lock portion, or the second lock portion of the database, wherein the third lock portions are more frequently updated than the second lock portion, and wherein more frequently updated values are distributed across the first lock portion and one or more third lock portions to avoid lock contention.
2. The method of claim 1 , wherein associating the first lock with the first lock portion of the database includes associating the first lock based on a frequency of updates to values in the first lock portion.
3. The method of claim 1 , wherein the first predetermined update aggregation rule is based on the type of values contained within the first column of the database.
4. The method of claim 1 , wherein the first predetermined update aggregation rule is satisfied when a threshold amount of time has passed or a threshold number of updates have been accumulated.
5. The method of claim 1 ,
wherein the accumulated plurality of first updates includes multiple updates to one particular value, and
wherein the accumulated plurality of first updates includes updates to multiple values.
6. A data processing system comprising:
a non-transitory computer-readable storage medium storing a set of computer-readable instructions configured to cause the data processing system to:
store a database that includes a plurality of database columns that each include a plurality of entries each having a respective value, wherein the plurality of database columns includes a first column;
associate a first lock with a first lock portion of the database;
associate a second lock with a second lock portion of the database, wherein the second lock portion does not overlap the first lock portion of the database and wherein the first lock portion of the database is more frequently updated than the second lock portion of the database;
accumulate a plurality of first updates, each first update being an update to a value in the first lock portion of the database, prior to acquiring the first lock, until a first predetermined update aggregation rule is satisfied;
acquire, in response to determining that the first predetermined update aggregation rule is satisfied, the first lock associated with the first lock portion of the database and then update the database by applying the accumulated first updates to the first lock portion of the database;
accumulate a plurality of second updates, each second update being an update to a value in the second lock portion of the database, prior to acquiring the second lock, until a second predetermined update aggregation rule is satisfied;
acquire, in response to determining that the second predetermined update aggregation rule is satisfied, the second lock associated with the second lock portion of the database and then update the database by applying the accumulated second updates to the second lock portion of the database; and
associate one or more third locks with respective third lock portions of the database, wherein the third lock portions do not overlap each other, the first lock portion, or the second lock portion of the database, wherein the third lock portions are more frequently updated than the second lock portion, and wherein more frequently updated values are distributed across the first lock portion and one or more third lock portions to avoid lock contention.
7. The system of claim 6 , wherein associating the first lock with the first lock portion of the database includes associating the first lock based on a frequency of updates to values in the first lock portion.
8. The system of claim 6 , wherein the first predetermined update aggregation rule is based on the type of values contained within the first column of the database.
9. The system of claim 6 , wherein the first predetermined update aggregation rule is satisfied when a threshold amount of time has passed or a threshold number of updates have been accumulated.
10. The system of claim 6 ,
wherein the accumulated plurality of first updates includes multiple updates to one particular value, and
wherein the accumulated plurality of first updates includes updates to multiple values.
11. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
storing a database that includes a plurality of database columns that each include a plurality of entries each having a respective value, wherein the plurality of database columns includes a first column;
associating a first lock with a first lock portion of the database;
associating a second lock with a second lock portion of the database, wherein the second lock portion does not overlap the first lock portion of the database and wherein the first lock portion of the database is more frequently updated than the second lock portion of the database;
accumulating a plurality of first updates, each first update being an update to a value in the first lock portion of the database, prior to acquiring the first lock, until a first predetermined update aggregation rule is satisfied;
in response to determining that the first predetermined update aggregation rule is satisfied, acquiring the first lock associated with the first lock portion of the database and then updating the database by applying the accumulated first updates to the first lock portion of the database;
accumulating a plurality of second updates, each second update being an update to a value in the second lock portion of the database, prior to acquiring the second lock, until a second predetermined update aggregation rule is satisfied;
in response to determining that the second predetermined update aggregation rule is satisfied, acquiring the second lock associated with the second lock portion of the database and then updating the database by applying the accumulated second updates to the second lock portion of the database; and
associating one or more third locks with respective third lock portions of the database, wherein the third lock portions do not overlap each other, the first lock portion, or the second lock portion of the database, wherein the third lock portions are more frequently updated than the second lock portion, and wherein more frequently updated values are distributed across the first lock portion and one or more third lock portions to avoid lock contention.