Method and apparatus for query and analysis
A data handling method combines search capabilities with analytical functionality. The invention provides advantages when dealing with structured documents (such as electronic catalogs, XML documents, text documents, HTML documents, Internet documents, etc.) and other data stored in a computer system. Various embodiments include simplified ways to express search/analysis requests of a data set and also to express results to such requests.
1. A method of selecting desired documents from a document collection comprising:
reading by a query engine executing on a computer system a first query expression, said first query expression comprising an expression of a first query to be performed by a query engine to identify a first subset of said collection, an analysis indicator comprising an identification of at least one of a plurality of analysis functions to be performed on said first subset by an analysis engine executing on the computer system, and one or more parameters for the analysis function;
running by the query execution engine the first query from the first query expression on the document collection to identify the first subset;
reading by the analysis engine the analysis indicator of the analysis to be performed on said first subset and the one or more parameters; and
generating by the analysis engine an analysis output by performing said analysis function with the one or more parameters on said first subset, wherein the analysis output comprises a second query expression comprising at least an expression of a second query to be performed by the query engine.
2. The method of claim 1 wherein said first query expression comprises a plurality of terms, each term comprising a name/value pair.
3. The method of claim 1 wherein said first query expression further comprises an operator separating each of the terms.
4. The method of claim 3 wherein said operators comprise standard boolean operators, statistical operators such as match N of M, and threshold expressions allowing different weighting for different parts of an expression.
5. The method of claim 1 wherein said first query expression further comprises one or more indications of linguistic tools.
6. The method of claim 1 further comprising:
determining by the analysis engine if said first subset contains sufficient data; and if not, modifying by the analysis engine said first query expression using a fallback procedure to generate the second query expression.
7. The method of claim 2 further comprising:
reading by the query engine a redirection indicator indicating that said analysis results are to be evaluated as queries to the document set.
8. The method of claim 7 further comprising:
reading by the query engine the second query expression outside of said redirection indicator and combining said second query expression with one or more query expressions output by said analysis function.
9. The method of claim 1 further comprising:
reading by the analysis engine an analysis parameter indicating a desired number of second query expressions to be returned by said analysis function.
10. The method of claim 2 wherein said plurality of analysis functions includes a rollup function that returns said indications of said N subsets of said first subset, each of said N subsets including documents that contain at least one of the top N most frequent values from an indicated field expression.
11. The method of claim 10 wherein said plurality of analysis functions includes a categorize function that returns said indications of said N subsets of said first subset, each of said N subsets including documents identified by determining patterns of name/value pairs that tend to occur together.
12. The method of claim 11 wherein said plurality of analysis functions includes a partition function that returns said indications of said N subsets of said first subset based on the number of the documents that appear in more than one of said N subsets and the number of the documents in each of said N subsets.
13. The method of claim 1 , wherein the first query expression further comprises at least one of a formatting clause defining the analysis output and a logging clause defining additional client-side information to be inserted into logs of the analysis output.
14. A method of selecting desired documents from a document collection comprising:
reading a first query expression, said first query expression comprising an expression of a first query to be performed by a query engine to identify a first subset of said collection and an analysis indicator comprising an identification of at least one of a plurality of analysis functions to be performed on said first subset;
running the first query from the first query expression on the document collection to identify the first subset;
reading the analysis indicator of the analysis to be performed on said first subset; and
generating an analysis output by performing said analysis function on said first subset, wherein the analysis output comprises a second query expression and wherein generating the second query expression comprises determining if said first subset contains sufficient data; and if not, modifying said first query using a fallback procedure to generate the second query expression.
15. The method of claim 14 wherein said first query expression comprises a plurality of terms, each term comprising a name/value pair.
16. The method of claim 14 wherein said first query expression further comprises an operator separating each of the terms.
17. The method of claim 16 wherein said operators comprise standard boolean operators, statistical operators such as match N of M, and threshold expressions allowing different weighting for different parts of an expression.
18. The method of claim 14 wherein said first query expression further comprises one or more indications of linguistic tools.
19. The method of claim 14 further comprising:
reading one or more parameters after said analysis indicator, said one or more parameters specifying modification of said analysis function;
reading a redirection indicator indicating that said analysis results are to be evaluated as queries to a document set; and
reading a second query expression outside of said redirection indicator and combining said second query expression with one or more query expressions output by said analysis function.
20. The method of claim 14 further comprising:
reading an analysis parameter indicating a desired number of second query expressions to be returned by said analysis function.
21. The method of claim 14 wherein said plurality of analysis functions includes a rollup function that returns said indications of said N subsets of said first subset, each of said N subsets including documents that contain at least one of the top N most frequent values from an indicated field expression.
22. The method of claim 21 wherein said plurality of analysis functions includes a categorize function that returns said indications of said N subsets of said first subset, each of said N subsets including documents identified by determining patterns of name/value pairs that tend to occur together.
23. The method of claim 22 wherein said plurality of analysis functions includes a partition function that returns said indications of said N subsets of said first subset based on the number of the documents that appear in more than one of said N subsets and the number of the documents in each of said N subsets.
24. Computer-readable memory having stored thereon instructions which, when executed by a computer, cause the computer to perform a method of selecting desired documents from a document collection, the method comprising:
reading by a query engine executing on the computer a first query expression, said first query expression comprising an expression of a first query to be performed by a query engine to identify a first subset of said collection, an analysis indicator comprising an identification of at least one of a plurality of analysis functions to be performed on said first subset by an analysis engine executing on the computer, and one or more parameters for the analysis function;
running by the query execution engine the first query from the first query expression on the document collection to identify the first subset;
reading by the analysis engine the analysis indicator of the analysis to be performed on said first subset and the one or more parameters; and
generating by the analysis engine an analysis output by performing said analysis function with the one or more parameters on said first subset, wherein the analysis output comprises a second query expression comprising at least an expression of a second query to be performed by the query engine.
25. The computer-readable memory of claim 24 wherein said first query expression comprises a plurality of terms, each term comprising a name/value pair.
26. The computer-readable memory of claim 24 wherein said first query expression further comprises an operator separating each of the terms.
27. The computer-readable memory of claim 26 wherein said operators comprise standard boolean operators, statistical operators such as match N of M, and threshold expressions allowing different weighting for different parts of an expression.
28. The computer-readable memory of claim 14 wherein said first query expression further comprises one or more indications of linguistic tools.
29. The computer-readable memory of claim 14 , wherein the method further comprises:
reading by the query engine a redirection indicator indicating that said analysis results are to be evaluated as queries to the document set.
30. The computer-readable memory of claim 29 , wherein the method further comprises:
reading by the query engine the second query expression outside of said redirection indicator and combining said second query expression with one or more query expressions output by said analysis function.
31. The computer-readable memory of claim 24 , wherein the method further comprises:
determining by the analysis engine if said first subset contains sufficient data; and if not, modifying said first query by the analysis engine using a fallback procedure to generate the second query.
32. The computer-readable memory of claim 24 , wherein the method further comprises:
reading by the analysis engine an analysis parameter indicating a desired number of second query expressions to be returned by said analysis function.
33. The computer-readable memory of claim 24 wherein said plurality of analysis functions includes a rollup function that returns said indications of said N subsets of said first subset, each of said N subsets including documents that contain at least one of the top N most frequent values from an indicated field expression.
34. The computer-readable memory of claim 33 wherein said plurality of analysis functions includes a categorize function that returns said indications of said N subsets of said first subset, each of said N subsets including documents identified by determining patterns of name/value pairs that tend to occur together.
35. The computer-readable memory of claim 34 wherein said plurality of analysis functions includes a partition function that returns said indications of said N subsets of said first subset based on the number of the documents that appear in more than one of said N subsets and the number of the documents in each of said N subsets.
36. Computer-readable memory having stored thereon instructions which, when executed by a computer, cause the computer to perform a method of selecting desired documents from a document collection, the method comprising:
reading a first query expression, said first query expression comprising an expression of a first query to be performed by a query engine to identify a first subset of said collection and an analysis indicator comprising an identification of at least one of a plurality of analysis functions to be performed on said first subset;
running the first query from the first query expression on the document collection to identify the first subset;
reading the analysis indicator of the analysis to be performed on said first subset; and
generating an analysis output by performing said analysis function on said first subset, wherein the analysis output comprises a second query expression and wherein generating the second query expression comprises determining if said first subset contains sufficient data; and if not, modifying said first query using a fallback procedure to generate the second query expression.
37. The computer-readable memory of claim 36 wherein said first query expression comprises a plurality of terms, each term comprising a name/value pair.
38. The computer-readable memory of claim 36 wherein said first query expression further comprises an operator separating each of the terms.
39. The computer-readable memory of claim 38 wherein said operators comprise standard boolean operators, statistical operators such as match N of M, and threshold expressions allowing different weighting for different parts of an expression.
40. The computer-readable memory of claim 36 wherein said first query expression further comprises one or more indications of linguistic tools.
41. The computer-readable memory of claim 36 , wherein the method further comprises:
reading a redirection indicator indicating that said analysis results are to be evaluated as queries to a document set; and
reading a second query expression outside of said redirection indicator and combining said second query expression with one or more query expressions output by said analysis function.
42. The computer-readable memory of claim 36 , wherein the method further comprises:
reading an analysis parameter indicating a desired number of second query expressions to be returned by said analysis function.
43. The computer-readable memory of claim 36 wherein said plurality of analysis functions includes a rollup function that returns said indications of said N subsets of said first subset, each of said N subsets including documents that contain at least one of the top N most frequent values from an indicated field expression.
44. The computer-readable memory of claim 43 wherein said plurality of analysis functions includes a categorize function that returns said indications of said N subsets of said first subset, each of said N subsets including documents identified by determining patterns of name/value pairs that tend to occur together.
45. The computer-readable memory of claim 44 wherein said plurality of analysis functions includes a partition function that returns said indications of said N subsets of said first subset based on the number of the documents that appear in more than one of said N subsets and the number of the documents in each of said N subsets.