Storing nested complex data structures in a data store
A data processing method provides an improvement in speed of obtaining, in response to a computer search query from one or more computing devices, a particular compressed data chunk that may include a particular data value that is digitally stored in a large data file. The method comprises creating and digitally storing a digital data file comprising a first set of uncompressed metadata chunks; a second set of uncompressed metadata chunks; and a set of compressed data chunks; each chunk of the first and second sets of uncompressed metadata chunks comprising a set of bits; the set of compressed data chunks comprising a first compressed data chunk; the first compressed data chunk comprising a first set of values; the first set of uncompressed metadata chunks spaced evenly across the data file; the second set of uncompressed metadata chunks spaced evenly between a first metadata chunk of the first set of uncompressed metadata chunks and a second metadata chunk of the first set of uncompressed metadata chunks; the first metadata chunk comprising a first set of bits. The method further comprises setting a first subset of bits of the first set of bits to indicate a minimum value of the first set of values, setting a second subset of bits of the first set of bits to indicate a maximum value of the first set of values, and setting a third subset of bits of the first set of bits to indicate a first portion of the data file separating the first metadata chunk and the second metadata chunk. The method also comprises receiving the search query specifying the particular data value, reading the first metadata chunk, and, in response to determining that the particular data value is equal to or greater than the minimum value and equal to or less than the maximum value, sending the first compressed data chunk to the one or more computing devices.
1. A data processing method comprising:
creating and digitally storing a digital file of data comprising a first set of uncompressed metadata chunks, a second set of uncompressed metadata chunks, and a set of compressed data chunks, each chunk of the first and second sets of uncompressed metadata chunks comprising a set of bits, a first compressed data chunk of the set of compressed data chunks comprising a first set of values, a first metadata chunk of the first set of uncompressed metadata chunks comprising a first set of bits;
setting a first subset of bits of the first set of bits to indicate a minimum value of the first set of values, setting a second subset of bits of the first set of bits to indicate a maximum value of the first set of values, and setting a third subset of bits of the first set of bits to indicate a first portion of the digital file of data separating the first metadata chunk and a second metadata chunk of the first set of uncompressed metadata chunks;
receiving a search query specifying a particular data value;
reading the first metadata chunk;
in response to determining that the particular data value is between the minimum value and the maximum value, sending the first compressed data chunk to one or more computing devices.
2. The method of claim 1 , wherein the digital file of data further comprises location metadata for the first set of uncompressed metadata chunks at an end of the digital file of data.
3. The method of claim 1 , wherein the first set of uncompressed metadata chunks are spaced evenly across the digital file of data.
4. The method of claim 1 , wherein the second set of uncompressed metadata chunks are spaced evenly between the first metadata chunk of the first set of uncompressed metadata chunks and the second metadata chunk of the first set of uncompressed metadata chunks.
5. The method of claim 1 , the set of compressed data chunks comprising a second compressed data chunk, the second compressed data chunk comprising a second set of values, and the second metadata chunk comprising a second set of bits, the method further comprising:
setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values, setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values, and setting a third subset of bits of the first set of bits to indicate a second portion of the digital file of data separating the second metadata chunk and a third metadata chunk of the first set of uncompressed metadata chunks;
in response to determining that the particular data value is not equal to or greater than the minimum value of the first set of values or equal to or less than the maximum value of the first set of values:
skipping the first portion of the digital file of data;
reading the second metadata chunk; and
in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.
6. The method of claim 1 , wherein the second set of uncompressed metadata chunks comprises a third metadata chunk, wherein the third metadata chunk comprises a second set of bits, wherein the third metadata chunk immediately precedes a second compressed data chunk, wherein the second compressed data chunk is a subset of the first compressed data chunk, and wherein the second compressed data chunk comprises a second set of values, the method further comprising:
in response to determining that the particular data value is equal to or greater than the minimum value of the first set of values and equal to or less than the maximum value of the first set of values, reading the third metadata chunk.
7. The method of claim 6 further comprising setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values.
8. The method of claim 6 further comprising setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values.
9. The method of claim 6 further comprising setting a third subset of bits of the second set of bits to indicate a second portion of the digital file of data separating the third metadata chunk and a fourth metadata chunk of the second set of uncompressed metadata chunks.
10. The method of claim 6 further comprising, in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.
11. The method of claim 1 , further comprising setting a fourth subset of bits of the first set of bits to indicate a number of values in the first set of values.
12. The method of claim 1 , wherein the first metadata chunk immediately precedes the set of compressed data chunks in the digital file of data.
13. The method of claim 1 wherein reading the first metadata chunk comprises performing a binary search of the first set of uncompressed metadata chunks and one or more linear searches of the second set of uncompressed metadata chunks.
14. A data processing system comprising:
one or more processors;
a memory storing instructions which, when executed by the one or more processors, causes performing:
creating and digitally storing a digital file of data comprising a first set of uncompressed metadata chunks, a second set of uncompressed metadata chunks, and a set of compressed data chunks, each chunk of the first and second sets of uncompressed metadata chunks comprising a set of bits, a first compressed data chunk of the set of compressed data chunks comprising a first set of values, a first metadata chunk of the first set of uncompressed metadata chunks comprising a first set of bits;
setting a first subset of bits of the first set of bits to indicate a minimum value of the first set of values, setting a second subset of bits of the first set of bits to indicate a maximum value of the first set of values, and setting a third subset of bits of the first set of bits to indicate a first portion of the digital file of data separating the first metadata chunk and a second metadata chunk of the first set of uncompressed metadata chunks;
receiving a search query specifying a particular data value;
reading the first metadata chunk;
in response to determining that the particular data value is between the minimum value and the maximum value, sending the first compressed data chunk to one or more computing devices.
15. The system of claim 14 , wherein the digital file of data further comprises location metadata for the first set of uncompressed metadata chunks at an end of the digital file of data.
16. The system of claim 14 , wherein the first set of uncompressed metadata chunks are spaced evenly across the digital file of data and the second set of uncompressed metadata chunks are spaced evenly between the first metadata chunk of the first set of uncompressed metadata chunks and the second metadata chunk of the first set of uncompressed metadata chunks.
17. The system of claim 14 , the set of compressed data chunks comprising a second compressed data chunk, the second compressed data chunk comprising a second set of values, and the second metadata chunk comprising a second set of bits, wherein the instructions, when executed by the one or more processors, further cause performance of:
setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values, setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values, and setting a third subset of bits of the first set of bits to indicate a second portion of the digital file of data separating the second metadata chunk and a third metadata chunk of the first set of uncompressed metadata chunks;
in response to determining that the particular data value is not equal to or greater than the minimum value of the first set of values or equal to or less than the maximum value of the first set of values:
skipping the first portion of the digital file of data;
reading the second metadata chunk; and
in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.
18. The system of claim 14 , wherein the second set of uncompressed metadata chunks comprises a third metadata chunk, wherein the third metadata chunk comprises a second set of bits, wherein the third metadata chunk immediately precedes a second compressed data chunk, wherein the second compressed data chunk is a subset of the first compressed data chunk, wherein the second compressed data chunk comprises a second set of values, and wherein the instructions, when executed by the one or more processors, further cause performance of:
in response to determining that the particular data value is equal to or greater than the minimum value of the first set of values and equal to or less than the maximum value of the first set of values, reading the third metadata chunk.
19. The system of claim 18 wherein the instructions, when executed by the one or more processors, further cause performance of setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values.
20. The system of claim 19 wherein the instructions, when executed by the one or more processors, further cause performance of:
setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values;
setting a third subset of bits of the second set of bits to indicate a second portion of the digital file of data separating the third metadata chunk and a fourth metadata chunk of the second set of uncompressed metadata chunks;
in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.