Matching documents against monitors
View Patent ↗Techniques and tools are described for matching documents against monitors. An index can be generated from a plurality of monitors, where the index represents the query logic of the plurality of monitors. The index can be searched using the documents as search queries. The searching can comprise matching the documents against the monitors using the query logic represented in the index. An index can be distributed to a plurality of computing devices to be searched at the plurality of computing devices, where each computing device searches a subset of a plurality of documents against the full index. Searching at the plurality of computing devices can be performed in parallel, and results can be aggregated at a central location.
1. A method, implemented at least in part by a computing device, for matching documents against monitors, the method comprising:
receiving, by the computing device, a plurality of monitors, wherein each monitor includes full query logic and one or more terms;
generating, by the computing device, an index from the plurality of monitors, the index representing full query logic for the plurality of monitors, wherein the index stores the plurality of monitors using, at least in part, an inverted index, the inverted index including entries for respective terms from the plurality of monitors, wherein an entry for a respective term includes a representation of the full query logic of each monitor having the respective term; and
storing, by the computing device, the index for the plurality of monitors, wherein the index is searchable using documents as queries to perform matching against the plurality of monitors.
2. The method of claim 1 , wherein the index stores the full query logic for each monitor of the plurality of monitors.
3. The method of claim 1 , wherein the representation of the full query logic of each monitor is a pointer to the full query logic for each monitor.
4. The method of claim 1 , further comprising:
receiving a plurality of documents;
searching the index using the plurality of documents as queries to determine matches between the plurality of documents and the plurality of monitors using the full query logic represented by the index, wherein a document matches a monitor when the document is satisfied by the full query logic of the monitor; and
returning results of the searching.
5. The method of claim 4 , wherein the searching the index comprises:
for each document of the plurality of documents:
determining which monitors, of the plurality of monitors, are satisfied by the document using the full query logic represented by the index.
6. The method of claim 4 , wherein the searching the index comprises:
for a document of the plurality of documents:
for each term of a plurality of terms in the document:
when the term is present in the index:
saving an indication that the term is present for each monitor containing the term; and
for each monitor containing the term, determining whether the monitor has been satisfied using the full query logic associated with the monitor.
7. The method of claim 6 , wherein the indication is saved in a cache list, wherein the cache list comprises entries for monitors for which at least one term has been found, wherein each entry comprises:
a unique identifier of the monitor;
a list of found terms for the monitor; and
an indication of whether the full query logic for the monitor has been satisfied.
8. The method of claim 6 , wherein the indication is saved in a cache list, wherein the cache list comprises entries for monitors for which at least one term has been found, wherein each entry comprises:
a unique identifier of the monitor;
a list of found terms for the monitor;
an indication of whether the full query logic for the monitor has been satisfied;
an indication of whether the monitor contains a “not” term; and
an indication of whether a “not” term has been found for the monitor.
9. The method of claim 5 , wherein a respective monitor further includes a user associated with the monitor, and the results of the searching are returned to the user when a respective document of the plurality of documents matches the respective monitor.
10. A search system for matching documents against monitors, the search system comprising:
a processing unit;
memory; and
one or more computer-readable storage media storing computer-executable instructions for causing the search system to perform operations comprising:
receiving a plurality of monitors, each monitor including full query logic and one or more terms;
generating an index from the plurality of monitors, the index representing full query logic for the plurality of monitors, wherein the index stores the plurality of monitors using, at least in part, an inverted index, the inverted index including entries for respective terms from the plurality of monitors, wherein an entry for a respective term includes a representation of the full query logic of each monitor having the respective term; and
storing the index for the plurality of monitors, wherein the index is searchable using documents as queries to perform matching against the plurality of monitors.
11. The search system of claim 10 , wherein the index comprises, for each monitor of the plurality of monitors, at least one of:
the full query logic for the monitor; or
a pointer to the full query logic for the monitor.
12. The search system of claim 10 , the operations further comprising:
receiving a plurality of documents;
searching the index using the plurality of documents as queries to determine matches between the plurality of documents and the plurality of monitors using the full query logic represented by the index, wherein a document matches a monitor when the document is satisfied by the full query logic of the monitor; and
returning results of the searching.
13. The search system of claim 12 , wherein the searching the index comprises:
for each document of the plurality of documents:
determining which monitors, of the plurality of monitors, are satisfied by the document using the full query logic represented by the index.
14. The search system of claim 12 , wherein the searching the index comprises:
for a document of the plurality of documents:
for each term of a plurality of terms in the document:
when the term is present in the index:
saving an indication that the term is present for each monitor containing the term; and
for each monitor containing the term, determining whether the monitor has been satisfied using the full query logic associated with the monitor.
15. The search system of claim 14 , wherein the indication is saved in a cache list, wherein the cache list comprises entries for monitors for which at least one term has been found, wherein each entry comprises:
a unique identifier of the monitor;
a list of found terms for the monitor; and
an indication of whether the full query logic for the monitor has been satisfied.
16. The search system of claim 14 , wherein the indication is saved in a cache list, wherein the cache list comprises entries for monitors for which at least one term has been found, wherein each entry comprises:
a unique identifier of the monitor;
a list of found terms for the monitor;
an indication of whether the full query logic for the monitor has been satisfied;
an indication of whether the monitor contains a “not” term; and
an indication of whether a “not” term has been found for the monitor.
17. The method of claim 13 , wherein a respective monitor further includes a user associated with the monitor, and the results of the searching are returned to the user when a respective document of the plurality of documents satisfies the respective monitor.
18. A method, implemented at least in part by a computer, for matching documents against monitors, the method comprising:
providing, by the computer, an index for storage at each of a plurality of computing devices, wherein the index is generated from a plurality of monitors, each monitor of the plurality of monitors including full query logic and one or more terms, the index represents full query logic for the plurality of monitors, and the index stores the plurality of monitors using, at least in part, an inverted index, the inverted index including entries for respective terms from the plurality of monitors, wherein an entry for a respective term includes a representation of the full query logic of each monitor having the respective term; and
receiving, by the computer, results of searching the index from each of the plurality of computing devices, wherein each computing device of the plurality of computing devices searches the index using a subset of a plurality of documents, and wherein each computing device of the plurality of computing devices searches the index using a different subset of the plurality of documents.
19. The method of claim 18 , further comprising:
receiving the plurality of monitors; and
generating the index from the plurality of monitors.
20. The method of claim 18 , wherein each computing device performs operations comprising:
searching the index, stored at the computing device, using its respective subset of the plurality of documents as queries to determine matches between the subset of the plurality of documents and the plurality of monitors using the full query logic represented by the index; and
returning results of the searching.
21. The method of claim 18 , wherein the plurality of documents are stored using a distributed file system, and wherein the subsets of the plurality of documents are accessible by the plurality of computing devices via the distributed file system.
22. The method of claim 18 , further comprising:
providing, by the computer, the plurality of documents for storage, wherein the plurality of documents are accessible by the plurality of computing devices.
23. The method of claim 18 , further comprising:
outputting results of the searching.
24. The method of claim 18 , wherein the method implements a distributed architecture in which the full index is stored at each of the plurality of computing devices, and in which the plurality of documents is segregated such that a separate subset of the plurality of documents is searched in parallel at each of the computing devices using the full index.
25. The method of claim 18 , wherein the index comprises, for each monitor of the plurality of monitors, at least one of:
the full query logic for the monitor; or
a pointer to the full query logic for the monitor.