Inverted indexes for accelerating analytics queries
The disclosed embodiments provide a system that processes data. During operation, the system obtains a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics. Next, the system creates, in a data segment comprising the records, an inverted index for a column in the records based on a cardinality of the column. Finally, the system compresses the inverted index based on a jump value associated with record identifiers in the column.
1. A computer-implemented method for processing data, comprising:
obtaining a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics;
creating, in a data segment comprising the records, an inverted index for a column in the records based on a cardinality of the column; and
compressing the inverted index based on a jump value associated with record identifiers in the column, which comprises:
a computer determining the jump value based on a threshold associated with compressing the inverted index; and
the computer including a record identifier in the compressed inverted index if a difference between the record identifier and a consecutive record identifier in the inverted index is greater than the jump value;
wherein the threshold is associated with a proportion of the record identifiers to be included in the compressed inverted index.
2. The computer-implemented method of claim 1 , further comprising:
enabling querying of the column using the compressed inverted index and a forward index of the column.
3. The computer-implemented method of claim 2 , wherein querying of the column is associated with alternating between the compressed inverted index and the forward index.
4. The computer-implemented method of claim 1 , wherein creating the inverted index for the column based on the cardinality of the column comprises:
creating the inverted index for the column if the column is associated with a standard cardinality or a high cardinality; and
omitting the inverted index for the column if the column is associated with a low cardinality.
5. The computer-implemented method of claim 1 , wherein the set of records comprises at least one of:
newly generated records of the one or more metrics and the at least one dimension; and
older records of the one or more metrics and the at least one dimension.
6. The computer-implemented method of claim 1 , wherein the column is an unsorted column.
7. A system for processing data, comprising:
a data queue configured to provide a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics; and
a segment-creation apparatus configured to:
create, in a data segment comprising the records, an inverted index for a column in the records based on a cardinality of the column; and
compress the inverted index based on a jump value associated with record identifiers in the column, which comprises:
determining the jump value based on a threshold associated with compressing the inverted index; and
including a record identifier in the compressed inverted index if a difference between the record identifier and a consecutive record identifier in the inverted index is greater than the jump value;
wherein the threshold is associated with a proportion of the record identifiers to be included in the compressed inverted index.
8. The system of claim 7 , further comprising:
a query-processing apparatus configured to enable querying of the column using the compressed inverted index and a forward index of the column.
9. The system of claim 8 , wherein querying of the column is associated with alternating between the compressed inverted index and the forward index.
10. The system of claim 7 , wherein creating the inverted index for the column based on the cardinality of the column comprises:
creating the inverted index for the column if the column is associated with a standard cardinality or a high cardinality; and
omitting the inverted index for the column if the column is associated with a low cardinality.
11. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for processing data, the method comprising:
obtaining a data segment comprising a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics;
creating, in a data segment comprising the records, an inverted index for a column in the records based on a cardinality of the column; and
compressing the inverted index based on a jump value associated with record identifiers in the column, which comprises:
determining the jump value based on a threshold associated with compressing the inverted index; and
including a record identifier in the compressed inverted index if a difference between the record identifier and a consecutive record identifier in the inverted index is greater than the jump value;
wherein the threshold is associated with a proportion of the record identifiers to be included in the compressed inverted index.
12. The computer-readable storage medium of claim 11 , the method further comprising:
enabling querying of the column using the compressed inverted index and a forward index of the column.
13. The computer-readable storage medium of claim 12 , wherein querying of the column is associated with alternating between the compressed inverted index and the forward index.
14. The computer-readable storage medium of claim 11 , wherein creating the inverted index for the column based on the cardinality of the column comprises:
creating the inverted index for the column if the column is associated with a standard cardinality or a high cardinality; and
omitting the inverted index for the column if the column is associated with a low cardinality.