Granted Patent B2

US 10,394,814 · App. 15/632,105 · Granted Aug 27, 2019

Abstract

Claims (47)

1. A data processing method providing an improvement in speed of obtaining, in response to a computer search query from one or more computing devices, a particular compressed data chunk that may include a particular data value that is digitally stored in a large data file, the method comprising the steps of:

creating and digitally storing a digital file of sorted data comprising a first set of uncompressed metadata chunks, a second set of uncompressed metadata chunks, a set of compressed data chunks, and location metadata for the first set of metadata chunks at an end of the digital file; each chunk of the first and second sets of uncompressed metadata chunks comprising a set of bits; the set of compressed data chunks comprising a first compressed data chunk; the first compressed data chunk comprising a first set of values; the first set of uncompressed metadata chunks spaced evenly across the data file; the second set of uncompressed metadata chunks spaced evenly between a first metadata chunk of the first set of uncompressed metadata chunks and a second metadata chunk of the first set of uncompressed metadata chunks; the first metadata chunk comprising a first set of bits;

setting a first subset of bits of the first set of bits to indicate a minimum value of the first set of values, setting a second subset of bits of the first set of bits to indicate a maximum value of the first set of values, and setting a third subset of bits of the first set of bits to indicate a first portion of the data file separating the first metadata chunk and the second metadata chunk;

receiving the search query specifying the particular data value;

reading the first metadata chunk; in response to determining that the particular data value is equal to or greater than the minimum value and equal to or less than the maximum value, sending the first compressed data chunk to the one or more computing devices.

2. The method of claim 1 , the set of compressed data chunks comprising a second compressed data chunk, the second compressed data chunk comprising a second set of values, and the second metadata chunk comprising a second set of bits, the method further comprising:

setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values, setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values, and setting a third subset of bits of the first set of bits to indicate a second portion of the data file separating the second metadata chunk and a third metadata chunk of the first set of metadata chunks;

in response to determining that the particular data value is not equal to or greater than the minimum value of the first set of values or equal to or less than the maximum value of the first set of values:

skipping the first portion of the data file;

reading the second metadata chunk; and

in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.

3. The method of claim 1 , wherein the second set of uncompressed metadata chunks comprises a third metadata chunk, wherein the third metadata chunk comprises a second set of bits, wherein the third metadata chunk immediately precedes a second compressed data chunk, wherein the second compressed data chunk is a subset of the first compressed data chunk, and wherein the second compressed data chunk comprises a second set of values, the method further comprising:

in response to determining that the particular data value is equal to or greater than the minimum value of the first set of values and equal to or less than the maximum value of the first set of values, reading the third metadata chunk.

4. The method of claim 3 further comprising setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values.

5. The method of claim 3 further comprising setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values.

6. The method of claim 3 further comprising setting a third subset of bits of the second set of bits to indicate a second portion of the data file separating the third metadata chunk and a fourth metadata chunk of the second set of uncompressed metadata chunks.

7. The method of claim 3 further comprising, in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.

8. The method of claim 1 , further comprising setting a fourth subset of bits of the first set of bits to indicate a number of values in the first set of values.

9. The method of claim 1 , wherein the first metadata chunk immediately precedes the set of compressed data chunks in the data file.

10. The method of claim 1 wherein reading the first metadata chunk comprises performing a binary search of the first set of uncompressed metadata chunks and one or more linear searches of the second set of uncompressed metadata chunks.

11. A data processing system providing an improvement in speed of obtaining, in response to a computer search query from one or more computing devices, a particular compressed data chunk that may include a particular data value that is digitally stored in a large data file, comprising:

one or more processors;

one or more non-transitory computer-readable storage media storing one or more sequences of instructions which when executed by the one or more processors cause performing:

creating and digitally storing a digital file of sorted data comprising a first set of uncompressed metadata chunks, a second set of uncompressed metadata chunks, a set of compressed data chunks, and location metadata for the first set of metadata chunks at an end of the digital file; each chunk of the first and second sets of uncompressed metadata chunks comprising a set of bits; the set of compressed data chunks comprising a first compressed data chunk; the first compressed data chunk comprising a first set of values; the first set of uncompressed metadata chunks spaced evenly across the data file; the second set of uncompressed metadata chunks spaced evenly between a first metadata chunk of the first set of uncompressed metadata chunks and a second metadata chunk of the first set of uncompressed metadata chunks; the first metadata chunk comprising a first set of bits;

setting a first subset of bits of the first set of bits to indicate a minimum value of the first set of values, setting a second subset of bits of the first set of bits to indicate a maximum value of the first set of values, and setting a third subset of bits of the first set of bits to indicate a first portion of the data file separating the first metadata chunk and the second metadata chunk;

receiving the search query specifying the particular data value;

reading the first metadata chunk;

in response to determining that the particular data value is equal to or greater than the minimum value and equal to or less than the maximum value, sending the first compressed data chunk to the one or more computing devices.

12. The system of claim 11 , the set of compressed data chunks comprising a second compressed data chunk, the second compressed data chunk comprising a second set of values, and the second metadata chunk comprising a second set of bits, the system further comprising sequences of instructions which when executed using the one or more processors cause:

setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values, setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values, and setting a third subset of bits of the first set of bits to indicate a second portion of the data file separating the second metadata chunk and a third metadata chunk of the first set of metadata chunks;

in response to determining that the particular data value is not equal to or greater than the minimum value of the first set of values or equal to or less than the maximum value of the first set of values:

skipping the first portion of the data file;

reading the second metadata chunk; and

in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.

13. The system of claim 11 , wherein the second set of uncompressed metadata chunks comprises a third metadata chunk, wherein the third metadata chunk comprises a second set of bits, wherein the third metadata chunk immediately precedes a second compressed data chunk, wherein the second compressed data chunk is a subset of the first compressed data chunk, and wherein the second compressed data chunk comprises a second set of values, the system further comprising sequences of instructions which when executed using the one or more processors cause:

in response to determining that the particular data value is equal to or greater than the minimum value of the first set of values and equal to or less than the maximum value of the first set of values, reading the third metadata chunk.

14. The system of claim 13 further comprising sequences of instructions which when executed using the one or more processors cause setting a first subset of bits of the second set of bits to indicate a minimum value of the second set of values, or setting a second subset of bits of the second set of bits to indicate a maximum value of the second set of values.

15. The system of claim 13 the system further comprising sequences of instructions which when executed using the one or more processors cause setting a third subset of bits of the second set of bits to indicate a second portion of the data file separating the third metadata chunk and a fourth metadata chunk of the second set of uncompressed metadata chunks.

16. The system of claim 13 further comprising sequences of instructions which when executed using the one or more processors cause, in response to determining that the particular data value is equal to or greater than the minimum value of the second set of values and equal to or less than the maximum value of the second set of values, sending the second compressed data chunk to the one or more computing devices.

17. The system of claim 11 , the system further comprising sequences of instructions which when executed using the one or more processors cause setting a fourth subset of bits of the first set of bits to indicate a number of values in the first set of values.

18. The system of claim 11 , wherein the first metadata chunk immediately precedes the set of compressed data chunks in the data file.

19. The system of claim 11 , the system further comprising sequences of instructions which when executed using the one or more processors cause reading the first metadata chunk by performing a binary search of the first set of uncompressed metadata chunks and one or more linear searches of the second set of uncompressed metadata chunks.

20. One or more non-transitory computer-readable storage media storing one or more sequences of instructions which when executed by the one or more processors cause performing a method providing an improvement in speed of obtaining, in response to a computer search query from one or more computing devices, a particular compressed data chunk that may include a particular data value that is digitally stored in a large data file, the method comprising the steps of:

creating and digitally storing a digital file of sorted data comprising a first set of uncompressed metadata chunks, a second set of uncompressed metadata chunks, a set of compressed data chunks, and location metadata for the first set of metadata chunks at an end of the digital file; each chunk of the first and second sets of uncompressed metadata chunks comprising a set of bits; the set of compressed data chunks comprising a first compressed data chunk; the first compressed data chunk comprising a first set of values; the first set of uncompressed metadata chunks spaced evenly across the data file; the second set of uncompressed metadata chunks spaced evenly between a first metadata chunk of the first set of uncompressed metadata chunks and a second metadata chunk of the first set of uncompressed metadata chunks; the first metadata chunk comprising a first set of bits;

setting a first subset of bits of the first set of bits to indicate a minimum value of the first set of values, setting a second subset of bits of the first set of bits to indicate a maximum value of the first set of values, and setting a third subset of bits of the first set of bits to indicate a first portion of the data file separating the first metadata chunk and the second metadata chunk;

receiving the search query specifying the particular data value;

reading the first metadata chunk; in response to determining that the particular data value is equal to or greater than the minimum value and equal to or less than the maximum value, sending the first compressed data chunk to the one or more computing devices.

Assignments (8)