Use of stream-oriented log data structure for full-text search oriented inverted index metadata
Systems and methods are disclosed for efficiently indexing stream data to facilitate full-text search of the stream data. A stream comprises a plurality of intervals of log data records. An interval of log data records are indexed. The index and log data records for the interval are written to an indexed stream data file. The index for each interval contains pointers to the terms in the log data records for the interval. After a number of intervals of index and log data records have been written, a merge operation can merge the number of intervals of index into a single merged index. The merged index and intervals of log data records are written to the indexed data stream file. A full-text search index is generated by traversing and merging the interval indexes for the data stream.
1. A computer-implemented method performed by a server, for efficiently generating a full-text search index for a data stream of records, the method comprising:
for each of a plurality of predetermined intervals of time:
receiving a plurality of data stream records;
generating an index of terms for the plurality of data stream records for the interval;
writing the index of terms for the plurality of data stream records for the interval, and the plurality of data stream records for the interval, into a single file termed an indexed data stream file;
in response to determining that the data stream records for a threshold number of predetermined time intervals have been indexed, generating the full-text search index for the plurality of time intervals by merging the index of terms from one or more of the plurality of intervals of time in the indexed data stream file in an ordered manner.
2. The method of claim 1 , wherein the index of terms for each of the plurality of intervals of time is generated in response to receiving a database query for the data stream of records.
3. The method of claim 1 , wherein the full-text search index is generated in response to a database query containing a database action, the method further comprising:
performing the database action against the data stream of records for a subset of the intervals of time, in accordance with the query.
4. The method of claim 3 , wherein the database action comprises one of: search, update, read, or delete.
5. The method of claim 1 , wherein the index of terms for each interval in the plurality of intervals references the terms in the plurality of data stream records indexed for the interval by a pointer.
6. The method of claim 1 , further comprising:
after a predetermined number of intervals of time:
merging the indexes from the indexed data streams files in the predetermined number of intervals, thereby generating a merged index for the predetermined number of intervals;
writing the merged index and the plurality of records from each of the indexed data stream files in the predetermined number of intervals into a single file containing the merged index and the plurality of records from each of the indexed data stream files in the predetermined number of intervals; and
marking for deletion, the indexed data stream file for each interval of the predetermined number of intervals.
7. A non-transitory computer readable medium, programmed with executable instructions that, when executed by a processing system, perform operations for efficiently generating a full-text search index for a data stream of records, the operations comprising:
for each of a plurality of predetermined intervals of time:
receiving a plurality of data stream records;
generating an index of terms for the plurality of data stream records for the interval;
writing the index of terms for the plurality of data stream records for the interval, and the plurality of data stream records for the interval, into a single file termed an indexed data stream file;
in response to determining that the data stream records for a threshold number of predetermined time intervals have been indexed, generating the full-text search index for the plurality of time intervals by merging the index of terms from one or more of the plurality of intervals of time in the indexed data stream file in an ordered manner.
8. The medium of claim 7 , wherein the index of terms for each of the plurality of intervals of time is generated in response to receiving a database query for the data stream of records.
9. The medium of claim 7 , wherein the full-text search index is generated in response to a database query containing a database action, the operations further comprising:
performing the database action against the data stream of records for a subset of the intervals of time, in accordance with the query.
10. The medium of claim 9 , wherein the database action comprises one of: search, update, index, or delete.
11. The medium of claim 7 , wherein the index of terms for each interval in the plurality of intervals references the terms in the plurality of data stream records indexed for the interval by a pointer.
12. The medium of claim 7 , the operations further comprising:
after a predetermined number of intervals of time:
merging the indexes from the indexed data stream files in the predetermined number of intervals, thereby generating a merged index for the predetermined number of intervals;
writing the merged index and the plurality of records from each of the indexed data stream files in the predetermined number of intervals into a single file containing the merged index and the plurality of records from each of the indexed data stream files in the predetermined number of intervals; and
marking for deletion, the indexed data stream file for each of the predetermined number of intervals.
13. A processing system, comprising a hardware processor coupled to a memory programmed with executable instructions, that when executed by the processing system, perform operations for efficiently generating a full-text search index for a data stream of records, the operations comprising:
for each of a plurality of predetermined intervals of time:
receiving a plurality of data stream records;
generating an index of terms for the plurality of data stream records for the interval;
writing the index of terms for the plurality of data stream records for the interval, and the plurality of data stream records for the interval, into a single file termed an indexed data stream file;
in response to determining that the data stream records for a threshold number of predetermined time intervals have been indexed, generating the full-text search index for the plurality of time intervals by merging the index of terms from one or more of the plurality of intervals of time in the indexed data stream file in an ordered manner.
14. The system of claim 13 , wherein the index of terms for each of the plurality of intervals of time is generated in response to receiving a database query for the data stream of records.
15. The system of claim 13 , wherein the full-text search index is generated in response to a database query containing a database action, the operations further comprising:
performing the database action against the data stream of records for a subset of the intervals of time, in accordance with the query.
16. The system of claim 15 , wherein the database action comprises one of: search, update, index, or delete.
17. The system of claim 13 , wherein the index of terms for each interval in the plurality of intervals references the terms in the plurality of data stream records indexed for the interval by a pointer.
18. The system of claim 13 , the operations further comprising:
after a predetermined number of intervals of time:
merging the indexes from the indexed data stream files in the predetermined number of intervals, thereby generating a merged index for the predetermined number of intervals;
writing the merged index and the plurality of records from each of the indexed data stream files in the predetermined number of intervals into a single file containing the merged index and the plurality of records from each of the indexed data stream files in the predetermined number of intervals; and
marking for deletion, the indexed data stream file for each of the predetermined number of intervals.