INDEXING AND SEQUENTIALLY STORING VARIABLE-LENGTH DATA TO FACILITATE REVERSE READING
A system, method, and apparatus are provided for indexing and sequentially storing variable-length data in a manner that facilitates reverse reading. Each entry stored in a log file, database, or other repository includes a data record having a fixed number of keys, a key offset corresponding to each key, and size metadata identifying a size of the data record (and possibly the key offsets). Each key offset is an offset to another entry (e.g., the matching key offset of the entry) whose data record features the same value for the corresponding key. An index identifies, for each given value of each key, an index offset to a first entry (e.g., the most recently stored entry) that has the given value for that key. Retrieving records matching a particular key value therefore simply involves following the corresponding index offset and then some number of key offsets.
1 . A method of indexing and sequentially storing variable-length data records, the method comprising:
obtaining a first data record having one or more keys with corresponding values;
for each key, retrieving from an index an index offset to a previously stored data record having the corresponding value;
storing the first data record;
adjacent to the first data record, storing key offsets derived from the retrieved index offsets; and
for each key, updating the index offset to reference the first data record instead of the previously stored data record.
2 . The method of claim 1 , wherein the index comprises:
for each of the one or more keys, at least one value; and
for each value of each of the one or more keys, an associated index offset to a stored data record having the value.
3 . The method of claim 1 , wherein:
the updated index offsets are offsets to a start of the first data record.
4 . The method of claim 1 , wherein:
for each key, said retrieving from the index an index offset to a previously stored data record having the corresponding value includes retrieving a first index offset for a first key;
said storing the retrieved index offsets as key offsets includes storing a first key offset comprising the first index offset; and
the updated first index offset is an offset to the first key offset.
5 . The method of claim 1 , further comprising:
(a) receiving a request for one or more data records having a specified value for a specified key;
(b) retrieving from the index a first index offset to a data record comprising the specified value for the specified key;
(c) accessing the data record;
(d) reading a key offset associated with the data record and corresponding to the specified key, wherein the key offset identifies a next data record comprising the specified value for the specified key;
(e) accessing the next data record; and
(f) repeating (d) and (e) until the request is satisfied.
6 . The method of claim 1 , wherein:
the index offsets are absolute offsets; and
the key offsets are relative offsets.
7 . The method of claim 1 , wherein:
the index offsets are absolute offsets;
the key offsets are relative offsets; and
storing a given key offset derived from a given retrieved index offset comprises converting the absolute offset of the given retrieved index offset to a relative offset from the given key offset.
8 . The method of claim 1 , further comprising:
determining a combined length of the data record and the key offsets;
storing the combined length;
determining the size of the combined length; and
when the size of the combined length is greater than a threshold, storing one additional byte following the combined length, wherein:
the most significant bit of the one additional byte is set to 1; and
remaining bits of the one additional byte identify the size of the combined length.
9 . An apparatus for indexing and sequentially storing variable-length data records, the apparatus comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
obtain a first data record having one or more keys with corresponding values;
for each key, retrieve from an index an index offset to a previously stored data record having the corresponding value;
store the first data record;
adjacent to the first data record, store key offsets derived from the retrieved index offsets; and
for each key, update the index offset to reference the first data record instead of the previously stored data record
10 . The apparatus of claim 9 , wherein the index comprises:
for each of the one or more keys, at least one value; and
for each value of each of the one or more keys, an associated index offset to a stored data record having the value.
11 . The apparatus of claim 9 , wherein:
the updated index offsets are offsets to a start of the first data record.
12 . The apparatus of claim 9 , wherein:
for each key, said retrieving from the index an index offset to a previously stored data record having the corresponding value includes retrieving a first index offset for a first key;
said storing the retrieved index offsets as key offsets includes storing a first key offset comprising the first index offset; and
the updated first index offset is an offset to the first key offset.
13 . The apparatus of claim 9 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
(a) receive a request for one or more data records having a specified value for a specified key;
(b) retrieve from the index a first index offset to a data record comprising the specified value for the specified key;
(c) access the data record;
(d) read a key offset associated with the data record and corresponding to the specified key, wherein the key offset identifies a next data record comprising the specified value for the specified key;
(e) access the next data record; and
(f) repeat (d) and (e) until the request is satisfied.
14 . The apparatus of claim 9 , wherein:
the index offsets are absolute offsets; and
the key offsets are relative offsets.
15 . The apparatus of claim 9 , wherein:
the index offsets are absolute offsets;
the key offsets are relative offsets; and
storing a given key offset derived from a given retrieved index offset comprises converting the absolute offset of the given retrieved index offset to a relative offset from the given key offset.
16 . The apparatus of claim 9 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
determine a combined length of the data record and the key offsets;
store the combined length;
determine the size of the combined length; and
when the size of the combined length is greater than a threshold, store one additional byte following the combined length, wherein:
the most significant bit of the one additional byte is set to 1; and
remaining bits of the one additional byte identify the size of the combined length.
17 . A system for indexing and sequentially storing variable-length data records, comprising:
at least one processor; and
a data storage module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
obtain a first data record having one or more keys with corresponding values;
for each key, retrieve from an index an index offset to a previously stored data record having the corresponding value;
store the first data record;
adjacent to the first data record, store key offsets derived from the retrieved index offsets; and
for each key, update the index offset to reference the first data record instead of the previously stored data record.
18 . The system of claim 17 , wherein the non-transitory computer-readable medium of the data storage module further stores instructions that, when executed, cause the system to:
(a) receive a request for one or more data records having a specified value for a specified key;
(b) retrieve from the index a first index offset to a data record comprising the specified value for the specified key;
(c) access the data record;
(d) read a key offset associated with the data record and corresponding to the specified key, wherein the key offset identifies a next data record comprising the specified value for the specified key;
(e) access the next data record; and
(f) repeat (d) and (e) until the request is satisfied.
19 . The system of claim 17 , wherein:
the index offsets are absolute offsets;
the key offsets are relative offsets; and
storing a given key offset derived from a given retrieved index offset comprises converting the absolute offset of the given retrieved index offset to a relative offset from the given key offset.
20 . The system of claim 17 , wherein the non-transitory computer-readable medium of the data storage module further stores instructions that, when executed, cause the system to:
determine a combined length of the data record and the key offsets;
store the combined length;
determine the size of the combined length; and
when the size of the combined length is greater than a threshold, store one additional byte following the combined length, wherein:
the most significant bit of the one additional byte is set to 1; and
remaining bits of the one additional byte identify the size of the combined length.