Row-order dependent dataframe workloads
A method includes receiving instructions to perform order-dependent DataFrame operation on data. In response to receiving the instructions, the framework analyzes the instructions to identify the order-dependent DataFrame operation, and generates an executable query corresponding to the identified order-dependent DataFrame operation. The framework executes the generated executable query on the data stored in the first database in the cloud data platform, creates a row position column that generates for each row a unique integer identifier, reproducible across multiple sessions and/or queries for the same underlying data to make the data accessible via positional indexing, and assigns each row of the first database a unique row index value based on the row position column and the row position column order. The framework orders a result from performing the order-dependent DataFrame operation based on the unique row index value assigned to each row of the first database, and returns the ordered result to the user.
1. A method comprising:
receiving, by at least one hardware processor, instructions to perform an order-dependent DataFrame operation on data stored in a first database in a cloud data platform, the instructions specified using code authored in a programming language for executing the order-dependent DataFrame operation within the cloud data platform;
analyzing the instructions to identify the order-dependent DataFrame operation;
generating an executable query corresponding to the identified order-dependent DataFrame operation;
executing the generated executable query on the data stored in the first database in the cloud data platform;
creating a row position column that generates a row position column order to make the data accessible via positional indexing;
assigning each row of the first database a unique row position value based on the row position column and the row position column order;
ordering a result from performing the order-dependent DataFrame operation based on the unique row position value assigned to each row of the first database; and
returning, to a user, the ordered result.
2. The method of claim 1 , wherein the order-dependent DataFrame operation comprises at least one of: displaying a specified number of rows, displaying a specified set of rows based on their indices, appending rows, or aggregating rows.
3. The method of claim 2 , wherein displaying the specified number of rows further comprises at least one of displaying the rows from a beginning of the first database or displaying the rows from an end of the first database.
4. The method of claim 1 , wherein assigning each row of the first database the unique row position value comprises:
determining an offset associated with a data file; and
assigning each row of the first database the unique row position value based on the offset of the data file and the unique row position value and a position of each row within the data file.
5. The method of claim 4 , wherein ordering the result comprises:
ordering a specified number of rows based on the unique row position value assigned to each row; and
modifying the generated executable query to include an ORDER BY clause corresponding to the unique row position value assigned to each row of the first database.
6. The method of claim 1 , wherein the order-dependent DataFrame operation comprises displaying rows from the first database based on specified row indices, and wherein ordering the result comprises ordering rows corresponding to the specified row indices based on the unique row position value assigned to each row.
7. The method of claim 1 , further comprising:
appending rows to the first database;
ordering the first database with the appended rows based on the unique row position value assigned to each row of the first database; and
assigning each of the appended rows with the unique row position value.
8. The method of claim 1 , further comprising:
performing an aggregation operation on the first database; and
ordering rows of an aggregation result based on values of grouping columns from the first database.
9. The method of claim 1 , further comprising:
joining the first database with a second database; and
ordering rows of a join result based on the unique row position value assigned to the rows of the first database and the rows of the second database.
10. The method of claim 9 , wherein ordering the rows of the join result further comprises:
prioritizing ordering the rows based on the unique row position value assigned to the first database over the unique row position value assigned to the second database.
11. A system comprising:
one or more hardware processors; and
at least one memory storing instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising:
receiving, by at least one hardware processor, instructions to perform an order-dependent DataFrame operation on data stored in a first database in a cloud data platform, the instructions specified using code authored in a programming language for executing the order-dependent DataFrame operation within the cloud data platform;
analyzing the instructions to identify the order-dependent DataFrame operation;
generating an executable query corresponding to the identified order-dependent DataFrame operation;
executing the generated executable query on the data stored in the first database in the cloud data platform;
creating a row position column that generates a row position column order to make the data accessible via positional indexing;
assigning each row of the first database a unique row position value based on the row position column and the row position column order;
ordering a result from performing the order-dependent DataFrame operation based on the unique row position value assigned to each row of the first database; and
returning, to a user, the ordered result.
12. The system of claim 11 , wherein the order-dependent DataFrame operation comprises at least one of: displaying a specified number of rows, displaying a specified set of rows based on their indices, appending rows, or aggregating rows.
13. The system of claim 12 , wherein displaying the specified number of rows further comprises at least one of displaying the rows from a beginning of the first database or displaying the rows from an end of the first database.
14. The system of claim 11 , wherein assigning each row of the first database the unique row position value comprises:
determining an offset associated with a data file; and
assigning each row of the first database the unique row position value based on the offset of the data file and the unique row position value and a position of each row within the data file.
15. The system of claim 14 , wherein ordering the result comprises:
ordering a specified number of rows based on the unique row position value assigned to each row; and
modifying the generated executable query to include an ORDER BY clause corresponding to the unique row position value assigned to each row of the first database.
16. The system of claim 11 , wherein the order-dependent DataFrame operation comprises displaying rows from the first database based on specified row indices, and wherein ordering the result comprises ordering rows corresponding to the specified row indices based on the unique row position value assigned to each row.
17. The system of claim 11 , the operations comprising:
appending rows to the first database;
ordering the first database with the appended rows based on the unique row position value assigned to each row of the first database; and
assigning each of the appended rows with the unique row position value.
18. The system of claim 11 , the operations comprising:
performing an aggregation operation on the first database; and
ordering rows of an aggregation result based on values of grouping columns from the first database.
19. The system of claim 11 , the operations comprising:
joining the first database with a second database; and
ordering rows of a join result based on the unique row position value assigned to the rows of the first database and the rows of the second database.
20. The system of claim 19 , wherein ordering the rows of the join result further comprises:
prioritizing ordering the rows based on the unique row position value assigned to the first database over the unique row position value assigned to the second database.
21. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by at least one hardware processor, instructions to perform an order-dependent DataFrame operation on data stored in a first database in a cloud data platform, the instructions specified using code authored in a programming language for executing the order-dependent DataFrame operation within the cloud data platform;
analyzing the instructions to identify the order-dependent DataFrame operation;
generating an executable query corresponding to the identified order-dependent DataFrame operation;
executing the generated executable query on the data stored in the first database in the cloud data platform;
creating a row position column that generates a row position column order to make the data accessible via positional indexing;
assigning each row of the first database a unique row position value based on the row position column and the row position column order;
ordering a result from performing the order-dependent DataFrame operation based on the unique row position value assigned to each row of the first database; and
returning, to a user, the ordered result.
22. The machine-storage medium of claim 21 , wherein the order-dependent DataFrame operation comprises at least one of: displaying a specified number of rows, displaying a specified set of rows based on their indices, appending rows, or aggregating rows.
23. The machine-storage medium of claim 22 , wherein displaying the specified number of rows further comprises at least one of displaying the rows from a beginning of the first database or displaying the rows from an end of the first database.
24. The machine-storage medium of claim 21 , wherein assigning each row of the first database the unique row position value comprises:
determining an offset associated with a data file; and
assigning each row of the first database the unique row position value based on the offset of the data file and the unique row position value and a position of each row within the data file.
25. The machine-storage medium of claim 24 , wherein ordering the result comprises:
ordering a specified number of rows based on the unique row position value assigned to each row; and
modifying the generated executable query to include an ORDER BY clause corresponding to the unique row position value assigned to each row of the first database.
26. The machine-storage medium of claim 21 , wherein the order-dependent DataFrame operation comprises displaying rows from the first database based on specified row indices, and wherein ordering the result comprises ordering rows corresponding to the specified row indices based on the unique row position value assigned to each row.
27. The machine-storage medium of claim 21 , wherein the operations comprise:
appending rows to the first database;
ordering the first database with the appended rows based on the unique row position value assigned to each row of the first database; and
assigning each of the appended rows with the unique row position value.
28. The machine-storage medium of claim 21 , wherein the operations comprise:
performing an aggregation operation on the first database; and
ordering rows of an aggregation result based on values of grouping columns from the first database.
29. The machine-storage medium of claim 21 , wherein the operations comprise:
joining the first database with a second database; and
ordering rows of a join result based on the unique row position value assigned to the rows of the first database and the rows of the second database.
30. The machine-storage medium of claim 29 , wherein ordering the rows of the join result further comprises:
prioritizing ordering the rows based on the unique row position value assigned to the first database over the unique row position value assigned to the second database.