Aggregating and emitting database activity record batches
Techniques for aggregating and emitting database activity record batches are described. Database activity records can be written to a shared memory queue and emitted to a destination using a remote procedure call (RPC). Individual database connection server processes can write client activity records to the queue. An activity monitor plugin in the database engine can monitor the audit records and aggregate the audit records into batches. Batches of audit records can be sent via RPC to their final or intermediate destination. Each instance host in a database service can include a client backend process configured to define how to submit audit records to shared memory. The activity monitor plugin can batch audit records in to messages and submit those messages via RPC to a security host manager and relaying response back to each relevant client backend.
1. A computer-implemented method comprising:
receiving a request to enable auditing of activity received by a database engine on an instance host in a database service in a provider network;
configuring the database engine to generate audit records that include a plurality of parameters;
allocating a shared memory by the database engine, the shared memory to store audit records from a plurality of client backend processes started by the database engine;
starting an activity monitor plugin on the database engine, the activity monitor plugin to monitor the shared memory based on one or more batch limits;
receiving a query by the database engine from a client device;
generating an audit record representing the query;
processing the query;
storing the audit record in the shared memory;
determining that the shared memory has reached a batch limit;
preparing a batch of audit records;
sending the batch of audit records to a security host manager;
receiving a response from the security host manager indicating the batch of audit records have been stored; and
returning the response and query results to the client device.
2. The computer-implemented method of claim 1 , wherein one or more batch limits include a time limit, batch size limit, or record number limit.
3. The computer-implemented method of claim 1 , wherein the batch of audit records is sent to the security host manager using a remote procedure call.
4. A computer-implemented method comprising:
generating, by an activity monitor of a database engine, an audit record representing a database query received from a client device;
processing the database query by the database engine on an instance host of a database service in a provider network to determine query results;
storing the audit record in a shared memory allocated by the database engine;
sending, by the activity monitor, a batch of audit records from the shared memory to a security host manager;
receiving, by the activity monitor, a response from the security host manager indicating that the batch of audit records have been stored; and
returning the query results to the client device.
5. The computer-implemented method of claim 4 , wherein the security host manager stores the audit records in the batch of audit records in a database associated with the database engine.
6. The computer-implemented method of claim 4 , wherein returning the query results to the client device further comprises:
returning the query results to the client device after the response has been received from the security host manager indicating that the batch of audit records have been stored.
7. The computer-implemented method of claim 4 , returning the query results to the client device further comprises:
returning the query results to the client device before the response has been received from the security host manager indicating that the batch of audit records have been stored.
8. The computer-implemented method of claim 4 , wherein the security host manager outputs an event stream including the audit records to an aggregating service.
9. The computer-implemented method of claim 4 , further comprising:
periodically generating, by the activity monitor, a heartbeat record; and
storing the heartbeat record in the shared memory, wherein the heartbeat record is included in the batch of audit records when it is sent to the security host manager.
10. The computer-implemented method of claim 4 , further comprising:
determining that the shared memory has reached a batch limit;
preparing the batch of audit records to be sent to the security host manager; and
updating a location in the shared memory with a value indicating processing status.
11. The computer-implemented method of claim 10 , wherein the batch limit includes at least one of a time limit, batch size limit, or record number limit.
12. The computer-implemented method of claim 4 , further comprising:
associating a timestamp with the audit record;
determining the response has not been received from the security host manager;
determining the timestamp has not elapsed; and
resending the batch of audit records to the security host manager.
13. A system comprising:
an aggregation service implemented by a first one or more electronic devices; and
a database service implemented by a second one or more electronic devices, the database service including instructions that upon execution cause the database service to:
generate, by an activity monitor of a database engine, an audit record representing a database query received from a client device;
process the database query by the database engine on an instance host of a database service in a provider network to determine query results;
store the audit record in a shared memory allocated by the database engine;
send, by the activity monitor, a batch of audit records from the shared memory to a security host manager;
receive, by the activity monitor, a response from the security host manager indicating that the batch of audit records have been stored; and
return the query results to the client device.
14. The system of claim 13 , wherein the security host manager stores the audit records in the batch of audit records in a database associated with the database engine.
15. The system of claim 13 , wherein to return the query results to the client device the instructions, when executed, further cause the database service to:
return the query results to the client device after the response has been received from the security host manager indicating that the batch of audit records have been stored.
16. The system of claim 13 , wherein to return the query results to the client device the instructions, when executed, further cause the database service to:
return the query results to the client device before the response has been received from the security host manager indicating that the batch of audit records have been stored.
17. The system of claim 13 , wherein the security host manager outputs an event stream including the audit records to an aggregating service.
18. The system of claim 13 , wherein the instructions, when executed, further cause the database service to:
periodically generate, by the activity monitor, a heartbeat record; and
store the heartbeat record in the shared memory, wherein the heartbeat record is included in the batch of audit records when it is sent to the security host manager.
19. The system of claim 13 , wherein the instructions, when executed, further cause the database service to:
determine that the shared memory has reached a batch limit;
prepare the batch of audit records to be sent to the security host manager; and
update a location in the shared memory with a value indicating processing status.
20. The system of claim 19 , wherein the batch limit includes at least one of a time limit, batch size limit, or record number limit.