Distributed stream-based database triggers
Information describing changes to a collection of items maintained by a database may be stored in a log file. The information in the log file may be converted into a stream of records describing the changes. The records may be directed to a computing node selected for performing a trigger function in response to the change, based on applying a hash function to a portion of the record, identifying a hash space associated with a value output by the hash function, and mapping from the hash space to the selected computing node.
1. A system comprising:
a database management system maintaining a collection of items corresponding to a hash space, wherein the database management system processes a first request to store an item in the collection of items by at least writing information indicative of the first request to a log file;
a plurality of computing nodes, including a first computing node, having access to a first set of instructions to be performed in response to the database management system processing the first request; and
a second computing node that at least:
receives the information indicative of the first request;
monitors resource utilization of computing nodes in the plurality of computing nodes;
selects the first computing node for performing the first set of instructions, based at least in part on resource utilization of the first computing node being less than an additional computing node of the plurality of computing nodes; and
transmits, to the first computing node based at least in part on the resource utilization of the first computing node being less than the additional computing node of the plurality of computing nodes, data causing the first computing node to perform the first set of instructions, wherein the data at least comprises the first set of instructions.
2. The system of claim 1 , wherein the second computing node at least:
monitors resource utilization of the first computing node; and
selects a computing node of the plurality of computing nodes for performing the first set of instructions, based at least in part on the resource utilization.
3. The system of claim 1 , wherein the second computing node at least:
monitors resource utilization of the first computing node, the resource utilization associated with the performance of the first set of instructions; and
determines to halt the performance of the first set of instructions based on the resource utilization.
4. The system of claim 1 , wherein the first set of instructions corresponds to a function of parameters comprising at least one of a primary key of the item, a prior value of the item, or a new value of the item.
5. The system of claim 1 , wherein capacity of the plurality of computing nodes may be scaled independently of the database management system.
6. A method for processing database trigger functions, the method comprising:
associating a plurality of computing nodes, including a first computing node, with a first set of instructions to be performed in response to a database management system receiving requests to store a plurality of items in a collection of items maintained by the database management system, the plurality of items and the first set of instructions corresponding to a first hash space;
receiving, from a repository of log data, information indicative of a first request, of a plurality of requests, to store an item in the collection of items;
causing the first computing node to obtain the first set of instructions based at least in part on determining that resources utilization of the first computing node is less than a second computing node of the plurality of computing nodes; and
causing the first set of instructions to be performed on the first computing node, based at least in part on determining that the resources utilization of the first computing node is less than the second computing node of the plurality of computing nodes.
7. The method of claim 6 , further comprising:
causing the first set of instructions to be performed on an additional computing node of the plurality of computing nodes based at least in part on resources utilized by performing the first set of instructions on the first computing node.
8. The method of claim 6 , further comprising:
halting performance of the first set of instructions when resources utilized by the performing the first set of instructions exceed a threshold amount.
9. The method of claim 6 , wherein the first set of instructions corresponds to a function of one or more parameters comprising at least one of a primary key of the item or information indicative of a change to the item.
10. The method of claim 6 , wherein the repository of log data comprises a subset of a log file of the database management system.
11. The method of claim 6 , further comprising:
storing information indicative of an association between the first hash space and a plurality of sets of instructions, the plurality of instructions including the first set of instructions; and
causing each of the plurality of sets of instructions associated with the database trigger functions associated with the first hash space to be executed in response to receiving the information indicative of the first request.
12. The method of claim 6 , further comprising:
causing a plurality of sets of instructions, including the first set of instructions, to be executed in response to receiving the information indicative of the first request.
13. A non-transitory computer-readable storage medium having stored thereon instructions that, upon execution by one or more computing devices, cause the one or more computing devices at least to:
associate a plurality of computing nodes, including a first computing node, with a first set of instructions associated with a collection of items maintained by a database, the first set of instructions received from a client of the database, the first set of instructions to be performed in response to the database receiving requests to store items in the collection of items;
receive information indicative of a first request, of a plurality of requests, to store an item in the collection of items;
cause the first computing node to obtain the first set of instructions based at least in part on determining that resources utilization of the first computing node is less than a second computing node of the plurality of computing nodes; and
cause the first set of instructions to be performed on the first computing node, based at least in part on determining that the resources utilization of the first computing node is less than the second computing node of the plurality of computing nodes.
14. The non-transitory computer-readable storage medium of claim 13 , comprising further instructions that, upon execution by the one or more computing devices, cause the one or more computing devices to at least:
transmit information indicative of performing the first set of instructions on an additional computing node of the plurality of computing nodes based at least in part on resources utilized by performing the first set of instructions on the first computing node.
15. The non-transitory computer-readable storage medium of claim 13 , comprising further instructions that, upon execution by the one or more computing devices, cause the one or more computing devices to at least:
determine to halt performance of the first set of instructions based at least in part on resources utilized by the performing the first set of instructions.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the first set of instructions corresponds to a function of one or more parameters comprising at least one of a primary key of the item or information indicative of a change to the item.
17. The non-transitory computer-readable storage medium of claim 13 , wherein items in the plurality of items corresponds to a first hash space.
18. The non-transitory computer-readable storage medium of claim 13 , comprising further instructions that, upon execution by the one or more computing devices, cause the one or more computing devices to at least:
store information indicative of an association between a first hash space and a plurality of sets of instructions, the plurality of instructions including the first set of instructions; and
cause each of the plurality of sets of instructions associated with the first hash space to be executed in response to receiving the information indicative of the first request.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the first hash space corresponds to a partition.
20. The non-transitory computer-readable storage medium of claim 13 , wherein the first set of instructions comprises a script for performing at least one of cross-region replication, data validation, or access pattern detection.