Function memoization in query processing system
A data platform that implements memoizable functions for database objects. The data platform detects a first execution of a memoizable function and generates a first key based on metadata of one or more database objects operated on by the memoizable function and generates a first result for the memoizable function based on the one or more database objects. The data platform detects a second execution of the memoizable function and generates a second key based on the metadata of the one or more database objects operated on by the memoizable function. When the first key and the second key are equal, the data platform reuses the first result of the memoizable function. When the first key and second key do not match, the data platform generates a second result for the second execution of the memoizable function.
1. A data platform comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
based on detecting a first execution of a memoizable function, performing the first execution by performing operations comprising:
generating a first key based on user identification of a first user having ownership of a first process performing the first execution and metadata of one or more database objects operated on by the memoizable function, the metadata comprising a modification history of the one or more database objects; and
generating a first result for the memoizable function based on the one or more database objects; and
based on detecting a second execution of the memoizable function, performing the second execution of the memoizable function by performing operations comprising:
generating a second key based on user identification of a second user having ownership of a second process performing the second execution and the metadata of the one or more database objects operated on by the memoizable function;
based on determining the second key is equal to the first key, reusing the first result of the first execution as a second result for the second execution; and
based on determining the second key is not equal to the first key, generating the second result for the second execution of the function based on the one or more database objects.
2. The data platform of claim 1 , wherein the instructions that cause the at least one processor to perform operations of storing the first key and the first result further cause the at least one processor to perform operations comprising:
based on determining that a memory requirement of the first result is below a memory storage limit, caching the first key and the first result; and
based on determining that the memory requirement of the first result is above a memory storage limit, not caching the first key and the first result.
3. The data platform of claim 1 , wherein the instructions that cause the at least one processor to perform operations of the first execution further cause the at least one processor to perform operations comprising:
launching a child job to perform the first execution.
4. The data platform of claim 3 , wherein the instructions that cause the at least one processor to perform operations of launching the child job further cause the at least one processor to perform operations comprising:
expanding a body of the memoizable function in the child job; and
replacing a function node with a result of the memoizable function.
5. The data platform of claim 1 , wherein the instructions that cause the at least one processor to perform operations of the second execution further cause the at least one processor to perform operations comprising:
launching a child job to perform the second execution within a security context of the data platform based on a security policy of the data platform.
6. The data platform of claim 5 , wherein the security policy comprises a security manager policy enforced by a security manager of the data platform.
7. The data platform of claim 5 , wherein the security policy comprises a sandbox policy enforced by a sandbox process of the data platform.
8. A computer-implemented method comprising:
based on detecting a first execution of a memoizable function, performing the first execution by performing operations comprising:
generating a first key based on user identification of a first user having ownership of a first process performing the first execution and metadata of one or more database objects operated on by the memoizable function, the metadata comprising a modification history of the one or more database objects; and
generating a first result for the memoizable function based on the one or more database objects; and
based on detecting a second execution of the memoizable function, performing the second execution of the memoizable function by performing operations comprising:
generating a second key based on user identification of a second user having ownership of a second process performing the second execution and the metadata of the one or more database objects operated on by the memoizable function;
based on determining the second key is equal to the first key, reusing the first result of the first execution as a second result for the second execution; and
based on determining the second key is not equal to the first key, generating the second result for the second execution of the function based on the one or more database objects.
9. The computer-implemented method of claim 8 , wherein the operations of storing the first key and the first result further comprise:
based on determining that a memory requirement of the first result is below a memory storage limit, caching the first key and the first result; and
based on determining that the memory requirement of the first result is above a memory storage limit, not caching the first key and the first result.
10. The computer-implemented method of claim 8 , wherein the operations of the first execution further comprise:
launching a child job to perform the first execution.
11. The computer-implemented method of claim 10 , wherein the operations of launching the child job further comprise:
expanding a body of the memoizable function in the child job; and
replacing a function node with a result of the memoizable function.
12. The computer-implemented method of claim 8 , wherein the operations of the second execution further comprise:
launching a child job to perform the second execution within a security context of a data platform based on a security policy of the data platform.
13. The computer-implemented method of claim 12 , wherein the security policy comprises a security manager policy enforced by a security manager of the data platform.
14. The computer-implemented method of claim 12 , wherein the security policy comprises a sandbox policy enforced by a sandbox process of the data platform.
15. A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a computer, cause the computer to perform operations comprising:
based on detecting a first execution of a memoizable function, performing the first execution by performing operations comprising:
generating a first key based on user identification of a first user having ownership of a first process performing the first execution and metadata of one or more database objects operated on by the memoizable function, the metadata comprising a modification history of the one or more database objects; and
generating a first result for the memoizable function based on the one or more database objects; and
based on detecting a second execution of the memoizable function, performing the second execution of the memoizable function by performing operations comprising:
generating a second key based on user identification of a second user having ownership of a second process performing the second execution and the metadata of the one or more database objects operated on by the memoizable function;
based on determining the second key is equal to the first key, reusing the first result of the first execution as a second result for the second execution; and
based on determining the second key is not equal to the first key, generating the second result for the second execution of the function based on the one or more database objects.
16. The non-transitory computer-storage medium of claim 15 , wherein the instructions that, when executed by the one or more processors of the computer, cause the computer to perform operations of storing the first key and the first result further cause the computer to perform operations comprising:
based on determining that a memory requirement of the first result is below a memory storage limit, caching the first key and the first result; and
based on determining that the memory requirement of the first result is above a memory storage limit, not caching the first key and the first result.
17. The non-transitory computer-storage medium of claim 15 , wherein the instructions that, when executed by the one or more processors cause the computer to perform operations of the first execution further cause the computer to perform operations comprising:
launching a child job to perform the first execution.
18. The non-transitory computer-storage medium of claim 17 , wherein the instructions that, when executed by the one or more processors cause the computer to perform operations of launching the child job further cause the computer to perform operations comprising:
expanding a body of the memoizable function in the child job; and
replacing a function node with a result of the memoizable function.
19. The non-transitory computer-storage medium of claim 15 , wherein the instructions that, when executed by the one or more processors cause the computer to perform operations of the second execution further cause the computer to perform operations comprising:
launching a child job to perform the second execution within a security context of a data platform based on a security policy of the data platform.
20. The non-transitory computer-storage medium of claim 19 , wherein the security policy comprises a security manager policy enforced by a security manager of the data platform.