Granted Patent B2

US 12,248,485 · App. 18/419,165 · Granted Mar 11, 2025

Inventors: Harsha S. Kapre (San Mateo, CA); Mark T. Keller (San Mateo, CA); Srinath Shankar (Belmont, CA); Kushan A Zaveri (North York, CA)

Assignee: Snowflake Inc.

G06F16/2471G06F16/24532G06F16/24561G06F16/256G06F16/258

Abstract

Claims (95)

1. A system comprising:

at least one hardware processor; and

at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:

retrieving query result information from a first computing node of a database system, the query result information including result data associated with a multi-stage execution of a query;

generating a plurality of result batches based on the query result information, the plurality of result batches corresponding to a plurality of execution stages of the multi-stage execution, and each result batch of the plurality of result batches including a retrieval function for a portion of the result data for a corresponding execution stage of the plurality of execution stages;

causing retrieval of the portion of the result data for each result batch of the plurality of result batches based on execution of the retrieval function by at least a second computing node of the database system; and

outputting the result data associated with the multi-stage execution of the query based on completion of the retrieval.

2. The system of claim 1 , wherein the portion of the result data corresponds to an execution stage of the multi-stage execution of the query, and wherein the instructions further cause the at least one hardware processor to perform operations comprising:

decoding the query result information to obtain storage information, the storage information indicating a storage location storing the result data associated with the multi-stage execution of the query.

3. The system of claim 2 , wherein the instructions further cause the at least one hardware processor to perform operations comprising:

encoding the retrieval function to include the storage information; and

causing retrieving of the result data from the storage location using the storage information.

4. The system of claim 2 , wherein the storage information is a uniform resource locator (URL) associated with the storage location.

5. The system of claim 2 , wherein the instructions further cause the at least one hardware processor to perform operations comprising:

decoding the query result information to further obtain authentication information associated with the storage location storing the result data.

6. The system of claim 5 , wherein the instructions further cause the at least one hardware processor to perform operations comprising:

encoding the retrieval function to include the storage information and the authentication information; and

causing retrieving of a plurality of data portions of the result data from the storage location using the storage information and the authentication information.

7. The system of claim 1 , wherein the instructions further cause the at least one hardware processor to perform operations comprising:

serializing the plurality of result batches before distribution to the at least a second computing node; and

causing deserialization of the plurality of result batches at the at least a second computing node before the retrieving.

8. The system of claim 1 , wherein the instructions further cause the at least one hardware processor to perform operations comprising:

decoding the query result information to obtain schema information, the schema information indicating a plurality of schemas associated with a plurality of data portions of the result data; and

parsing the plurality of data portions based on the plurality of schemas to obtain parsed result data; and

outputting the parsed result data.

9. The system of claim 1 , wherein the instructions further cause the at least one hardware processor to perform operations comprising:

distributing the plurality of result batches to a plurality of processing threads executing on the at least a second computing node;

causing parallel fetching and processing of corresponding portions of a plurality of data portions of the result data by the plurality of processing threads to generate processed data, using the plurality of result batches; and

causing retrieving of the processed data from the at least a second computing node.

10. The system of claim 1 , wherein the instructions further cause the at least one hardware processor to perform operations comprising:

obtaining a plurality of data portions of the result data from the at least a second computing node;

converting one or more of the plurality of data portions of the result data into a data table or a data object; and

providing the data table or the data object as output, in response to a data processing request associated with the multi-stage execution of the query.

11. A method comprising:

retrieving, by at least one hardware processor, query result information from a first computing node of a database system, the query result information including result data associated with a multi-stage execution of a query;

generating a plurality of result batches based on the query result information, the plurality of result batches corresponding to a plurality of execution stages of the multi-stage execution, and each result batch of the plurality of result batches including a retrieval function for a portion of the result data for a corresponding execution stage of the plurality of execution stages;

causing retrieval of the portion of the result data for each result batch of the plurality of result batches based on execution of the retrieval function by at least a second computing node of the database system; and

outputting the result data associated with the multi-stage execution of the query based on completion of the retrieval.

12. The method of claim 11 , wherein the portion of the result data corresponds to an execution stage of the multi-stage execution of the query, and the method further comprising:

decoding the query result information to obtain storage information, the storage information indicating a storage location storing the result data associated with the multi-stage execution of the query.

13. The method of claim 12 , further comprising:

encoding the retrieval function to include the storage information; and

causing retrieving of the result data from the storage location using the storage information.

14. The method of claim 12 , wherein the storage information is a uniform resource locator (URL) associated with the storage location.

15. The method of claim 12 , further comprising:

decoding the query result information to further obtain authentication information associated with the storage location storing the result data.

16. The method of claim 15 , further comprising:

encoding the retrieval function to include the storage information and the authentication information; and

causing retrieving of a plurality of data portions of the result data from the storage location using the storage information and the authentication information.

17. The method of claim 11 , further comprising:

serializing the plurality of result batches before distribution to the at least a second computing node; and

causing deserialization of the plurality of result batches at the at least a second computing node before the retrieving.

18. The method of claim 11 , further comprising:

decoding the query result information to obtain schema information, the schema information indicating a plurality of schemas associated with a plurality of data portions of the result data; and

parsing the plurality of data portions based on the plurality of schemas to obtain parsed result data; and

outputting the parsed result data.

19. The method of claim 11 , further comprising:

distributing the plurality of result batches to a plurality of processing threads executing on the at least a second computing node;

causing parallel fetching and processing of corresponding portions of a plurality of data portions of the result data by the plurality of processing threads to generate processed data, using the plurality of result batches; and

causing retrieving of the processed data from the at least a second computing node.

20. The method of claim 11 , further comprising:

obtaining a plurality of data portions of the result data from the at least a second computing node;

converting one or more of the plurality of data portions of the result data into a data table or a data object; and

providing the data table or the data object as output, in response to a data processing request associated with the multi-stage execution of the query.

21. A computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

retrieving query result information from a first computing node of a database system, the query result information including result data associated with a multi-stage execution of a query;

generating a plurality of result batches based on the query result information, the plurality of result batches corresponding to a plurality of execution stages of the multi-stage execution, and each result batch of the plurality of result batches including a retrieval function for a portion of the result data for a corresponding execution stage of the plurality of execution stages;

causing retrieval of the portion of the result data for each result batch of the plurality of result batches based on execution of the retrieval function by at least a second computing node of the database system; and

outputting the result data associated with the multi-stage execution of the query based on completion of the retrieval.

22. The computer-storage medium of claim 21 , wherein the portion of the result data corresponds to an execution stage of the multi-stage execution of the query, and the operations further comprising:

decoding the query result information to obtain storage information, the storage information indicating a storage location storing the result data associated with the multi-stage execution of the query.

23. The computer-storage medium of claim 22 , the operations further comprising:

encoding the retrieval function to include the storage information; and

causing retrieving of the result data from the storage location using the storage information.

24. The computer-storage medium of claim 22 , wherein the storage information is a uniform resource locator (URL) associated with the storage location.

25. The computer-storage medium of claim 22 , the operations further comprising:

decoding the query result information to further obtain authentication information associated with the storage location storing the result data.

26. The computer-storage medium of claim 25 , the operations further comprising:

encoding the retrieval function to include the storage information and the authentication information; and

causing retrieving of a plurality of data portions of the result data from the storage location using the storage information and the authentication information.

27. The computer-storage medium of claim 21 , the operations further comprising:

serializing the plurality of result batches before distribution to the at least a second computing node; and

causing deserialization of the plurality of result batches at the at least a second computing node before the retrieving.

28. The computer-storage medium of claim 21 , the operations further comprising:

decoding the query result information to obtain schema information, the schema information indicating a plurality of schemas associated with a plurality of data portions of the result data; and

parsing the plurality of data portions based on the plurality of schemas to obtain parsed result data; and

outputting the parsed result data.

29. The computer-storage medium of claim 21 , the operations further comprising:

distributing the plurality of result batches to a plurality of processing threads executing on the at least a second computing node;

causing parallel fetching and processing of corresponding portions of a plurality of data portions of the result data by the plurality of processing threads to generate processed data, using the plurality of result batches; and

causing retrieving of the processed data from the at least a second computing node.

30. The computer-storage medium of claim 21 , the operations further comprising:

obtaining a plurality of data portions of the result data from the at least a second computing node;

converting one or more of the plurality of data portions of the result data into a data table or a data object; and

providing the data table or the data object as output, in response to a data processing request associated with the multi-stage execution of the query.

Assignments (1)

Continuity (3)

References Cited (68)