Systems and methods for hybrid storage
Embodiments of the present disclosure are directed to a method for storing and processing data. The method includes identifying a database in a memory of a host device having one or more rows and one or more columns. A partition having a partition size is identified, and the one or more rows of the database is identified based on the partition size. The data stored in the one or more rows is converted into a column-based format, and the data is stored in a computational storage device in the column-based format. The computational storage device is configured to retrieve the data stored in the column-based format, in response to a query, and process the query based on the data.
1 . A method for storing and processing data, the method comprising:
identifying a database stored in a first memory of a host device, the database having one or more rows and one or more columns, the host device including a first processor coupled to the first memory;
identifying, by the first processor, a partition size;
based on identifying the partition size, identifying, by the first processor, the one or more rows of the database based on the partition size;
converting, by the first processor, data stored in the one or more rows, into a column-based format;
storing the data in a data structure having the partition size, wherein the data is stored in the data structure in the column-based format;
determining, by the first processor, a condition related to the partition size;
storing, by the first processor, second data in the data structure based on determining the condition related to the partition size; and
transmitting, by the first processor, a command to store the data structure in a storage device having a second processor and a second memory, wherein the data in the data structure is stored in the second memory in the column-based format, wherein in response to a query, the second processor of the storage device is configured to retrieve the data stored in the column-based format from the second memory, and process the query based on the data.
2 . The method of claim 1 , wherein the partition size is a multiple of a size of a database page.
3 . The method of claim 1 , wherein the partition size is based on a redundant array of Independent (RAID) stripe size.
4 . The method of claim 1 , wherein in the column-based format, values of one of the one or more columns for the one or more rows are stored contiguously in an address space.
5 . The method of claim 1 , wherein the converting of the data includes generating a database page storing the data in the column-based format.
6 . The method of claim 5 , wherein the database page includes one or more pointers to the one or more columns stored in the database page.
7 . The method of claim 5 further comprising:
encoding the database page, wherein the database page is included in the partition;
determining a second characteristic of the partition; and
generating a second database page based on determining the second characteristic, wherein the second database page stores data in a second row of the database, in the column-based format; and
including the second database page into the partition.
8 . The method of claim 5 further comprising:
encoding the database page, wherein the database page is included in the partition;
determining a second characteristic of the partition; and
based on determining the second characteristic, adding second data stored in a second row of the database, to the database page, in the column-based format.
9 . The method of claim 1 , wherein the database is generated by an application running in the host device, wherein the query is generated by the application.
10 . The method of claim 1 , wherein the second memory of the storage device includes a non-volatile memory.
11 . A system for storing and processing data, the system comprising:
a first processor; and
a first memory, wherein the first memory stores instructions that, when executed by the first processor, cause the first processor to:
identify a database stored in the first memory, the database having one or more rows and one or more columns;
identify a partition size;
based on identifying the partition size, identify the one or more rows of the database based on the partition size;
convert data stored in one or more columns of the one or more rows, into a column-based format, wherein the data stored in the column-based format is associated with the partition;
store the data in a data structure having the partition size, wherein the data is stored in the data structure in the column-based format;
determine a condition related to the partition size:
store second data in the data structure based on determining the condition related to the partition size; and
transmit a command to store the data structure in a storage device having a second processor and a second memory, wherein the data in the data structure is stored in the second memory in the column format, wherein in response to a query, the second processor of the storage device is configured to retrieve the data stored in the column format from the second memory, and process the query based on the data.
12 . The system of claim 11 , wherein the partition size is a multiple of a size of a database page.
13 . The system of claim 11 , wherein the partition size is based on a redundant array of Independent (RAID) stripe size.
14 . The system of claim 11 , wherein in the column-based format, values of one of the one or more columns for the one or more rows are stored contiguously in an address space.
15 . The system of claim 11 , wherein the instructions that cause the first processor to convert the data include instructions that cause the first processor to generate a database page storing the data in the column-based format.
16 . The system of claim 15 , wherein the database page includes one or more pointers to the one or more columns stored in the database page.
17 . The system of claim 15 , wherein the instructions further cause the first processor to:
encode the database page, wherein the database page is included in the partition;
determine a characteristic of the partition; and
generate a second database page based on the determining, wherein the second database page stores data in a second row of the database, in the column-based format; and
include the second database page into the partition.
18 . The system of claim 15 , wherein the instructions further cause the first processor to:
encode the database page, wherein the database page is included in the partition;
determine a characteristic of the partition; and
based on the determining, add second data stored in a second row of the database, to the database page, in the column-based format.
19 . The system of claim 11 , wherein the database is generated by an application executed by the first processor, wherein the query is generated by the application.
20 . The system of claim 11 , wherein the second memory of the storage device includes a non-volatile memory.