Telemetry for query processing memory usage
A method may include receiving, from a first thread of a first operator performing a first operation of a query, a first request for memory allocation or memory deallocation. In response to the first request, a first local data structure of the first thread may be accessed to identify the first operator associated with the first request. A first memory usage metric for the first thread of the first operator may be updated based on the first request. A third memory usage metric for the first operator may be determined based on the first memory usage metric and a second memory usage metric for a second thread of the first operator. A fourth memory usage metric for a second operator may be determined based on a fifth memory usage metric for a third thread used by the second operator to perform a second operation of the query.
1. A system, comprising:
at least one processor; and
at least one memory storing instructions which, when executed by the at least one processor, cause operations comprising:
receiving, from a first thread of a first operator performing a first operation associated with a query, a first request for memory allocation or memory deallocation;
responding to the first request by at least accessing a first local data structure of the first thread to identify the first operator associated with the first request;
updating, based at least on the first request, a first memory usage metric for the first thread of the first operator, wherein the updating of the first memory usage metric includes adding, to a running aggregate of a memory integral, a result of multiplying a size of a current memory allocation by a difference between a current timestamp and a timestamp of a last update;
determining, based at least on the first memory usage metric for the first thread of the first operator and a second memory usage metric for a second thread of the first operator, a third memory usage metric for the first operator; and
responding to the first request by allocating or deallocating at least a portion of a memory allocated to the query.
2. The system of claim 1 , wherein the operations further comprise:
receiving, from the second thread of the first operator, a second request for memory allocation or memory deallocation;
responding to the second request by at least accessing a second local data structure of the second thread to identify the first operator associated with the second request; and
updating, based at least on the second request, the second memory usage metric for the second thread of the first operator.
3. The system of claim 1 , wherein the operations further comprise:
receiving, from a third thread of a second operator performing a second operation associated with the query, a second request for memory allocation or memory deallocation;
responding to the second request by at least accessing a second local data structure of the third thread to identify the second operator associated with the second request;
updating, based at least on the second request, a fourth memory usage metric for the third thread of the second operator; and
determining, based at least on the fourth memory usage metric for the third thread of the second operator, a fifth memory usage metric for the second operator.
4. The system of claim 1 , wherein the first local data structure of the first thread includes a reference identifying the first operator to identify the first operator as being associated with the first request.
5. The system of claim 1 , wherein the first memory usage metric includes the memory integral corresponding to a memory usage over time.
6. The system of claim 5 , wherein the updating of the first memory usage metric includes updating a first value corresponding to the running aggregate of the memory integral, a second value corresponding to the size of the current memory allocation, and a third value corresponding to the timestamp of the last update to the running aggregate of the memory integral.
7. The system of claim 1 , wherein the first memory usage metric includes one or more of peak memory usage, minimum size of memory allocation, maximum size of memory allocation, average size of memory allocation, and total quantity of memory allocations.
8. The system of claim 1 , wherein the first local data structure comprises a thread-local storage (TLS) of the first thread.
9. The system of claim 1 , wherein the first thread and/or the second thread are executed in parallel with a third thread used by a second operator to perform a second operation associated with the query.
10. A computer-implemented method, comprising:
receiving, from a first thread of a first operator performing a first operation associated with a query, a first request for memory allocation or memory deallocation;
responding to the first request by at least accessing a first local data structure of the first thread to identify the first operator associated with the first request;
updating, based at least on the first request, a first memory usage metric for the first thread of the first operator, wherein the updating of the first memory usage metric includes adding, to a running aggregate of a memory integral, a result of multiplying a size of a current memory allocation by a difference between a current timestamp and a timestamp of a last update;
determining, based at least on the first memory usage metric for the first thread of the first operator and a second memory usage metric for a second thread of the first operator, a third memory usage metric for the first operator; and
responding to the first request by allocating or deallocating at least a portion of a memory allocated to the query.
11. The method of claim 10 , further comprising:
receiving, from the second thread of the first operator, a second request for memory allocation or memory deallocation;
responding to the second request by at least accessing a second local data structure of the second thread to identify the first operator associated with the second request; and
updating, based at least on the second request, the second memory usage metric for the second thread of the first operator.
12. The method of claim 10 , further comprising:
receiving, from a third thread of a second operator performing a second operation associated with the query, a second request for memory allocation or memory deallocation;
responding to the second request by at least accessing a second local data structure of the third thread to identify the second operator associated with the second request;
updating, based at least on the second request, a fourth memory usage metric for the third thread of the second operator; and
determining, based at least on the fourth memory usage metric for the third thread of the second operator, a fifth memory usage metric for the second operator.
13. The method of claim 10 , wherein the first local data structure of the first thread includes a reference identifying the first operator to identify the first operator as being associated with the first request.
14. The method of claim 10 , wherein the first memory usage metric includes one or more of peak memory usage, minimum size of memory allocation, maximum size of memory allocation, average size of memory allocation, and total quantity of memory allocations.
15. The method of claim 10 , wherein the first local data structure comprises a thread-local storage (TLS) of the first thread.
16. A non-transitory computer readable medium storing instructions, which when executed by at least one processor, result in operations comprising:
receiving, from a first thread of a first operator performing a first operation associated with a query, a first request for memory allocation or memory deallocation;
responding to the first request by at least accessing a first local data structure of the first thread to identify the first operator associated with the first request;
updating, based at least on the first request, a first memory usage metric for the first thread of the first operator, wherein the updating of the first memory usage metric includes adding, to a running aggregate of a memory integral, a result of multiplying a size of a current memory allocation by a difference between a current timestamp and a timestamp of a last update;
determining, based at least on the first memory usage metric for the first thread of the first operator and a second memory usage metric for a second thread of the first operator, a third memory usage metric for the first operator; and
responding to the first request by allocating or deallocating at least a portion of a memory allocated to the query.