Incremental maintenance of inverted indexes for approximate string matching
In embodiments of the disclosed technology, indexes, such as inverted indexes, are updated only as necessary to guarantee answer precision within predefined thresholds which are determined with little cost in comparison to the updates of the indexes themselves. With the present technology, a batch of daily updates can be processed in a matter of minutes, rather than a few hours for rebuilding an index, and a query may be answered with assurances that the results are accurate or within a threshold of accuracy.
1. A method for limiting updates of inverted indexes in a database, the method comprising:
calculating a first inverse document frequency for each n-gram in a plurality of strings;
calculating a first inverse document frequency related length for each string in the plurality of strings;
for each n-gram, creating an inverted list comprising the strings containing the n-gram;
sorting each inverted list by the first inverse document frequency related length of the strings in the inverted list;
receiving an update to the plurality of strings;
determining which n-grams are affected by the update;
calculating, based on the update, a second inverse document frequency of the affected n-grams;
determining which strings are affected by the update;
calculating, based on the update, a second inverse document frequency related length of the affected strings;
determining which inverted lists are affected by the update; and
for each affected inverted list:
calculating an error based on the second inverse document frequencies;
determining whether the error is less than a predefined error threshold; and
upon determining that the error is not less than the predefined error threshold, updating the affected inverted list; and
answering a query based on the updated affected inverted lists.
2. The method of claim 1 , wherein the update is an addition of a string.
3. The method of claim 1 , wherein the update is a deletion of a string.
4. The method of claim 1 , wherein the update is a modification of a string.
5. The method of claim 1 , wherein the error is based on a difference between the second inverse document frequency and the first inverse document frequency.
6. The method of claim 1 , wherein the predefined error threshold is determined such that no false dismissals of an answer occur in answering the query.
7. The method of claim 1 , wherein the predefined error threshold is determined such that a small number of false positive answers is introduced in answering the query.
8. A search engine database processing device comprising:
means for calculating a first inverse document frequency for each n-gram in a plurality of strings;
means for calculating a first inverse document frequency related length for each string in the plurality of strings;
means for, for each n-gram, creating an inverted list comprising the strings containing the n-gram;
means for sorting each inverted list by the first inverse document frequency related length of the strings in the inverted list;
means for receiving an update to the plurality of strings;
means for determining which n-grams are affected by the update;
means for calculating, based on the update, a second inverse document frequency of the affected n-grams;
means for determining which strings are affected by the update;
means for, for each affected inverted list:
calculating an error based on the second inverse document frequencies;
determining whether the error is less than a predefined error threshold; and
upon determining that the error is not less than the predefined error threshold, updating the affected inverted list; and
means for answering a query based on the updated affected inverted lists.
9. The device of claim 8 , wherein the update is an addition of a string.
10. The device of claim 8 , wherein the update is a deletion of a string.
11. The device of claim 8 , wherein the update is a modification of a string.
12. The device of claim 8 , wherein the error is based on a difference between the second inverse document frequency and the first inverse document frequency.
13. The device of claim 8 , wherein the predefined error threshold is determined such that no false dismissals of an answer occur answering the query.
14. The device of claim 8 , wherein the predefined error threshold is determined such that a small number of false positive answers is introduced in answering the query.