User defined function memoization
A data platform that implements memoizable functions for database objects. The data platform detects a first execution of a memoizable function and generates a first key based on metadata of one or more database objects operated on by the memoizable function and generates a first result for the memoizable function based on the one or more database objects. The data platform detects a second execution of the memoizable function and generates a second key based on the metadata of the one or more database objects operated on by the memoizable function. When the first key and the second key are equal, the data platform reuses the first result of the memoizable function. When the first key and second key do not match, the data platform generates a second result for the second execution of the memoizable function.
1 . A computer-implemented method comprising:
detecting a first execution of a user-defined function (UDF);
generating a first key based on metadata of one or more database objects operated on by the UDF, the metadata comprising a modification history of the one or more database objects;
generating a first result for the first execution of the UDF based on the one or more database objects;
storing the first key and the first result;
detecting a second execution of the UDF;
generating a second key based on the metadata of the one or more database objects;
based on determining the second key matches the first key, reusing the stored first result for the second execution of the UDF; and
based on determining the second key does not match the first key, generating a second result for the second execution of the UDF based on the one or more database objects.
2 . The computer-implemented method of claim 1 , wherein the operations of storing the first key and the first result further comprise:
based on determining that a memory requirement of the first result is below a memory storage limit, caching the first key and the first result; and
based on determining that the memory requirement of the first result is above a memory storage limit, not caching the first key and the first result.
3 . The computer-implemented method of claim 1 , wherein operations of the first execution comprise:
launching a child job to perform the first execution.
4 . The computer-implemented method of claim 3 , wherein operations of launching the child job comprise:
expanding a body of the UDF in the child job; and
replacing a function node with a result of the UDF.
5 . The computer-implemented method of claim 3 , wherein operations of launching the child job comprise:
instantiating a virtual machine and allocating computing resources for executing the UDF.
6 . The computer-implemented method of claim 3 , wherein operations of launching the child comprise:
performing the first execution within a security context of a data platform based on a security policy of the data platform.
7 . The computer-implemented method of claim 6 , wherein the security policy comprises a security manager policy enforced by a security manager of the data platform.
8 . The computer-implemented method of claim 7 , wherein the security policy comprises a sandbox policy enforced by a sandbox process of the data platform.
9 . The computer-implemented method of claim 1 , wherein the stored first result persists across multiple sessions.
10 . The computer-implemented method of claim 1 , wherein the modification history comprises timestamps associated with modifications to the one or more database objects.
11 . A machine comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
detecting a first execution of a user-defined function (UDF);
generating a first key based on metadata of one or more database objects operated on by the UDF, the metadata comprising a modification history of the one or more database objects;
generating a first result for the first execution of the UDF based on the one or more database objects;
storing the first key and the first result;
detecting a second execution of the UDF;
generating a second key based on the metadata of the one or more database objects;
based on determining the second key matches the first key, reusing the stored first result for the second execution of the UDF; and
based on determining the second key does not match the first key, generating a second result for the second execution of the UDF based on the one or more database objects.
12 . The machine of claim 11 , wherein the operations of storing the first key and the first result further comprise:
based on determining that a memory requirement of the first result is below a memory storage limit, caching the first key and the first result; and
based on determining that the memory requirement of the first result is above a memory storage limit, not caching the first key and the first result.
13 . The machine of claim 11 , wherein operations of the first execution comprise:
launching a child job to perform the first execution.
14 . The machine of claim 13 , wherein operations of launching the child job comprise:
expanding a body of the UDF in the child job; and
replacing a function node with a result of the UDF.
15 . The machine of claim 13 , wherein operations of launching the child job comprise:
instantiating a virtual machine and allocating computing resources for executing the UDF.
16 . The machine of claim 13 , wherein operations of launching the child comprise:
performing the first execution within a security context of a data platform based on a security policy of the data platform.
17 . The machine of claim 16 , wherein the security policy comprises a security manager policy enforced by a security manager of the data platform.
18 . The machine of claim 17 , wherein the security policy comprises a sandbox policy enforced by a sandbox process of the data platform.
19 . The machine of claim 11 , wherein the stored first result persists across multiple sessions.
20 . The machine of claim 11 , wherein the modification history comprises timestamps associated with modifications to the one or more database objects.
21 . A machine-storage medium storing instructions that, when executed by the machine, cause the machine to perform operations comprising:
detecting a first execution of a user-defined function (UDF);
generating a first key based on metadata of one or more database objects operated on by the UDF, the metadata comprising a modification history of the one or more database objects;
generating a first result for the first execution of the UDF based on the one or more database objects;
storing the first key and the first result;
detecting a second execution of the UDF;
generating a second key based on the metadata of the one or more database objects;
based on determining the second key matches the first key, reusing the stored first result for the second execution of the UDF; and
based on determining the second key does not match the first key, generating a second result for the second execution of the UDF based on the one or more database objects.
22 . The machine-storage medium of claim 21 , wherein the operations of storing the first key and the first result further comprise:
based on determining that a memory requirement of the first result is below a memory storage limit, caching the first key and the first result; and
based on determining that the memory requirement of the first result is above a memory storage limit, not caching the first key and the first result.
23 . The machine-storage medium of claim 21 , wherein operations of the first execution comprise:
launching a child job to perform the first execution.
24 . The machine-storage medium of claim 23 , wherein operations of launching the child job comprise:
expanding a body of the UDF in the child job; and
replacing a function node with a result of the UDF.
25 . The machine-storage medium of claim 23 , wherein operations of launching the child job comprise:
instantiating a virtual machine and allocating computing resources for executing the UDF.
26 . The machine-storage medium of claim 23 , wherein operations of launching the child comprise:
performing the first execution within a security context of a data platform based on a security policy of the data platform.
27 . The machine-storage medium of claim 26 , wherein the security policy comprises a security manager policy enforced by a security manager of the data platform.
28 . The machine-storage medium of claim 27 , wherein the security policy comprises a sandbox policy enforced by a sandbox process of the data platform.
29 . The machine-storage medium of claim 21 , wherein the stored first result persists across multiple sessions.
30 . The machine-storage medium of claim 21 , wherein the modification history comprises timestamps associated with modifications to the one or more database objects.