Using predicates in conditional transcoder for column store
A storage device is disclosed. The storage device may comprise storage for input encoded data. A controller may process read requests and write requests from a host computer on the data in the storage. An in-storage compute controller may receive a predicate from the host computer to be applied to the input encoded data. A transcoder may include an index mapper to map an input dictionary to an output dictionary, with one entry in the input dictionary mapped to an entry in the output dictionary, and another entry in the input dictionary mapped to a “don't care” entry in the output dictionary.
1 . A storage device, comprising:
storage for a first data and a second data, the first data including a first entry and a second entry, one of the first entry and the second entry applicable to the second data;
a processor to receive an instruction, the instruction to be applied to the first data, the instruction including a filtering data; and
a controller to generate a third entry and a fourth entry from the first entry and the second entry based at least in part on the filtering data,
wherein one of first entry and the second entry is different from the third entry or the fourth entry.
2 . The storage device according to claim 1 , wherein the controller includes at least one of a processor, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), or a General Purpose GPU (GPGPU).
3 . The storage device according to claim 1 , wherein the processor is operative to apply an acceleration function on a third data encoded using at least one of the third entry or the fourth entry.
4 . The storage device according to claim 3 , wherein:
the storage further stores a fourth data; and
the storage device is configured to generate the third data based at least in part on the fourth data, the third entry, and the fourth entry.
5 . The storage device according to claim 4 , wherein:
the fourth data uses a first encoding scheme;
the third data uses a second encoding scheme; and
the second encoding scheme is different from the first encoding scheme.
6 . The storage device according to claim 4 , further comprising:
a first buffer to store the first data;
a second buffer to store a third data, the third data based at least in part on the first data, the third entry, and the fourth entry;
a third buffer to store a fourth data, the fourth data based at least in part on a fifth data, the third entry, and the fourth entry; and
a rule evaluator to generate an output stream based at least in part on the third data in the second buffer, the fourth data in the third buffer, and at least one transcoding rule.
7 . The storage device according to claim 6 , wherein the at least one transcoding rule is based at least in part on the filtering data.
8 . The storage device according to claim 1 , wherein the data is stored in the storage in a columnar format.
9 . The storage device according to claim 8 , further comprising a column chunk processor to process a column chunk including the data.
10 . The storage device according to claim 9 , wherein the column chunk processor includes at least one of a processor, an FPGA, an ASIC, a GPU, or a GPGPU.
11 . A method, comprising:
receiving an instruction to apply to a first data, the instruction including a filtering data;
identifying a first entry in the first data covered by the instruction and a second entry in the first data not covered by the instruction, the first entry and the second entry applicable to a second data; and
generating a third data including a third entry and a fourth entry based at least in part on the first entry, the second entry, and the filtering data.
12 . The method according to claim 11 , wherein the data includes a column chunk stored in a columnar format.
13 . The method according to claim 11 , further comprising:
transcoding the second data into a fourth data based at least in part on the third data; and
outputting the fourth data.
14 . The method according to claim 13 , wherein:
transcoding the first data into the fourth data based at least in part on the third data includes:
receiving a first chunk from the first data;
determining that the first chunk is covered by the filtering data; and
generating a second chunk from the first chunk of data based at least in part on the third data; and
outputting the fourth data includes outputting the second chunk.
15 . The method according to claim 14 , further comprising generating transcoding rules to apply to the first data based at least in part on the filtering data.
16 . The method according to claim 13 , wherein outputting the fourth data includes outputting the fourth data to a controller of a storage device.
17 . The method according to claim 16 , wherein outputting the fourth data to the controller further includes outputting the third data to the controller.
18 . The method according to claim 17 , further comprising performing an acceleration function on the fourth data by the controller to produce a fifth data.
19 . An article, comprising a non-transitory storage medium, the non-transitory storage medium having stored thereon instructions that, when executed by a machine, result in:
receiving an instruction to apply to a first data, the instruction including a filtering data;
identifying a first entry in the first data covered by the instruction and a second entry in the first data not covered by the instruction, the first entry and the second entry applicable to a second data; and
generating a third data including a third entry and a fourth entry based at least in part on the first entry, the second entry, and the filtering data.
20 . The article according to claim 19 , wherein the data includes a column chunk stored in a columnar format.