Hardware-implemented file reader
A hardware-implemented file reader includes an interface, multiple hardware-implemented column readers and a hardware-implemented record reconstructor. The interface is configured to access a file including multiple records. The records store values in accordance with a nested structure that supports optional values and repeated values. The file is stored in a columnar format having multiple columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column to the nested structure of the records. Each column reader is configured to be assigned to a respective selected column, and to read and decompress both the values and the structure information from at least a portion of the selected column. The record reconstructor is configured to reconstruct one or more of the records from at least portions of the columns that are read by the column readers, and to output the reconstructed records.
1. A hardware-implemented file reader, comprising:
an interface, configured to access a file comprising multiple records, wherein the records store values in accordance with a structure that supports optional values and repeated values, and wherein the file is stored in a columnar format having multiple columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column to the nested structure of the records;
multiple hardware-implemented column readers, each column reader configured to be assigned to a respective selected column, and to read and decompress both the values and the structure information from at least a portion of the selected column, wherein at least a given column reader among the column readers comprises a hardware-implemented pipeline configured to incrementally decrypt, decompress and decode portions of the values or the structure information; and
a hardware-implemented record reconstructor, configured to reconstruct one or more of the records from at least portions of the columns that are read by the column readers, and to output the reconstructed records.
2. The file reader according to claim 1 , wherein the hardware-implemented pipeline comprises:
decryption logic configured to decrypt the values or structure information;
a first buffer configured to buffer the decrypted values or structure information produced by the decryption logic;
decompression logic configured to decompress the decrypted values or structure information buffered in the first buffer;
a second buffer configured to buffer the decompressed values or structure information produced by the decompression logic; and
a decoder configured to decode the decompressed values or structure information buffered in the second buffer.
3. The file reader according to claim 2 , wherein at least the given column reader, including the hardware-implemented pipeline including the first and second buffers, is implemented within an Integrated Circuit (IC).
4. The file reader according to claim 2 , wherein the second buffer is configured to apply backpressure to the decompression logic, and the first buffer is configured to apply backpressure to the decryption logic, thereby throttling a rate of readout from the selected column.
5. The file reader according to claim 1 , wherein the given column reader or the record reconstructor is configured to manipulate at least some of the decoded values or structure information.
6. The file reader according to claim 1 , wherein the given column reader or the record reconstructor comprises a hardware-implemented filter configured to filter the records based on one or both of (i) a criterion defined over one or more of the values, and (ii) a received query.
7. The file reader according to claim 1 , wherein the given column reader is configured to align at least some of the decompressed values with the corresponding decompressed structure information, before reading and decompressing subsequent values and subsequent structure information from the selected column.
8. A method for hardware-implemented file readout, comprising:
accessing a file using multiple hardware-implemented column readers, wherein the file comprises multiple records, wherein the records store values in accordance with a structure that supports optional values and repeated values, and wherein the file is stored in a columnar format having multiple columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column to the nested structure of the records;
assigning each column reader to a respective selected column, and reading and decompressing both the values and the structure information from at least a portion of the selected column, including, in at least a given column reader among the column readers, incrementally decrypting, decompressing and decoding portions of the values or the structure information using a hardware-implemented pipeline; and
using a hardware-implemented record reconstructor, reconstructing one or more of the records from at least portions of the columns that are read by the column readers, and outputting the reconstructed records.
9. The method according to claim 8 , wherein incrementally decrypting, decompressing and decoding portions of the values or the structure information comprises:
decrypting the values or structure information using decryption logic;
buffering the decrypted values or structure information produced by the decryption logic in a first buffer;
decompressing the decrypted values or structure information buffered in the first buffer using decompression logic;
buffering the decompressed values or structure information produced by the decompression logic in a second buffer; and
decoding the decompressed values or structure information buffered in the second buffer using a decoder.
10. The method according to claim 9 , wherein at least the given column reader, including the hardware-implemented pipeline including the first and second buffers, is implemented within an Integrated Circuit (IC).
11. The method according to claim 9 , further comprising applying backpressure to the decompression logic using the second buffer, and applying backpressure to the decryption logic using the first buffer, thereby throttling a rate of readout from the selected column.
12. The method according to claim 8 , further comprising, in the given column reader or in the record reconstructor, manipulating at least some of the decoded values or structure information.
13. The method according to claim 8 , further comprising, in the given column reader or in the record reconstructor, filtering the records using a hardware-implemented filter, based on one or both of (i) a criterion defined over one or more of the values, and (ii) a received query.
14. The method according to claim 8 , further comprising aligning at least some of the decompressed values with the corresponding decompressed structure information, before reading and decompressing subsequent values and subsequent structure information from the selected column.