Token based dynamic data indexing with integrated security
Semantic analysis to resolve ambiguous user input data with respect to a request for data includes identifying tokens based on the text string, identifying a tables in a database, wherein a token indicates a column from a table and a token indicates a column from another table, identifying candidate join paths for joining tables, obtaining respective candidate results using the candidate join paths, outputting combined candidate results including values from the respective candidate results, obtaining second user input data indicating a selected value, identifying the request for data as unambiguous and identifying a selected join path based on the selected value, and, in response to identifying the request for data as unambiguous, outputting data responsive to the request for data using the selected join path.
1. A method comprising:
obtaining, by an information retrieval system, from a database management system, enterprise data;
automatically generating, by the information retrieval system, enterprise data tokens representing the enterprise data;
automatically indexing, by the information retrieval system, the enterprise data tokens in an index structure of the information retrieval system;
obtaining, by the information retrieval system, first user input data including a natural language string expressing a request for data from the information retrieval system;
automatically generating, by the information retrieval system, a semantic representation of the natural language string, in a form that differs from a structured query language of the database management system, wherein generating the semantic representation includes:
traversing the index structure to match a portion of the natural language string to an enterprise data token from the enterprise data tokens; and
including the enterprise data token in the semantic representation;
automatically converting, by the information retrieval system, the semantic representation into a structured query language query expressing the request for data;
obtaining, by the information retrieval system, from the database management system, in response to the structured query language query, results data responsive to the request for data; and
automatically outputting, for presentation to a user, the results data.
2. The method of claim 1 , wherein indexing the enterprise data tokens includes:
including, in the index structure, a root node;
including, in the index structure, a first branch depending from the root node, the first branch representing a first symbol from the enterprise data token; and
including, in the index structure, a security bitmask for the enterprise data token, such that a security bitmask for the first branch at the root node is a hierarchical logical disjunction based on the security bitmask for the enterprise data token.
3. The method of claim 2 , wherein:
obtaining the first user input data includes obtaining a security bitmask for the first user input data; and
automatically generating the semantic representation includes:
determining that the first symbol matches a symbol from the portion of the natural language string; and
determining that a horizontal logical disjunction of a vertical logical conjunction of the security bitmask for the first branch and the security bitmask for the first user input data indicates authorization.
4. The method of claim 1 , wherein:
obtaining the first user input data includes obtaining the first user input data from a user device; and
automatically outputting the results data includes outputting the results data to the user device.
5. The method of claim 1 , wherein:
automatically indexing the enterprise data tokens includes:
including, in the index structure, data indicating an association between the enterprise data token and a first table stored in the database management system; and
including, in the index structure, data indicating an association between a second enterprise data token from the enterprise data tokens and a second table stored in the database management system;
automatically generating the semantic representation includes:
traversing the index structure to match a second portion of the natural language string to the second enterprise data token;
including the second enterprise data token in the semantic representation;
identifying a join path for joining data from the first table with data from the second table;
including data indicating the join path in the semantic representation; and
automatically converting the semantic representation includes:
including data indicating the join path in the structured query language query.
6. The method of claim 5 , wherein:
obtaining the enterprise data includes:
obtaining relationship data indicating a relationship between the first table and the second table stored; and
storing the relationship data in the information retrieval system; and
identifying the join path includes using the relationship data.
7. The method of claim 1 , wherein:
automatically indexing the enterprise data tokens includes:
including, in the index structure, data indicating an association between the enterprise data token and a column of a table stored in the database management system; and
automatically generating the semantic representation includes:
in response to determining, by a finite state machine of the information retrieval system, that the column is a measure column, including, in the semantic representation, data indicating an aggregation operation with respect to the measure column.
8. The method of claim 1 , wherein:
automatically indexing the enterprise data tokens includes:
including, in the index structure, data indicating an association between the enterprise data token and a first column of a first table stored in the database management system;
automatically generating the semantic representation includes:
identifying candidate joint paths, wherein identifying the candidate join paths includes:
identifying a first candidate join path for joining data from the first column with data from a second column from a second table stored in the database management system; and
identifying a second candidate join path for joining data from the first column with data from a third column from the second table;
obtaining second user input data indicating a join path from the candidate join paths;
including data indicating the join path in the semantic representation; and
automatically converting the semantic representation includes:
including data indicating the join path in the structured query language query.
9. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
obtaining, by an information retrieval system, from a database management system, enterprise data;
automatically generating, by the information retrieval system, enterprise data tokens representing the enterprise data;
automatically indexing, by the information retrieval system, the enterprise data tokens in an index structure of the information retrieval system;
obtaining, by the information retrieval system, first user input data including a natural language string expressing a request for data from the information retrieval system;
automatically generating, by the information retrieval system, a semantic representation of the natural language string, in a form that differs from a structured query language of the database management system, wherein generating the semantic representation includes:
traversing the index structure to match a portion of the natural language string to an enterprise data token from the enterprise data tokens; and
including the enterprise data token in the semantic representation;
automatically converting, by the information retrieval system, the semantic representation into a structured query language query expressing the request for data;
obtaining, by the information retrieval system, from the database management system, in response to the structured query language query, results data responsive to the request for data; and
automatically outputting, for presentation to a user, the results data.
10. The non-transitory computer-readable storage medium of claim 9 , wherein indexing the enterprise data tokens includes:
including, in the index structure, a root node;
including, in the index structure, a first branch depending from the root node, the first branch representing a first symbol from the enterprise data token; and
including, in the index structure, a security bitmask for the enterprise data token, such that a security bitmask for the first branch at the root node is a hierarchical logical disjunction based on the security bitmask for the enterprise data token.
11. The non-transitory computer-readable storage medium of claim 10 , wherein:
obtaining the first user input data includes obtaining a security bitmask for the first user input data; and
automatically generating the semantic representation includes:
determining that the first symbol matches a symbol from the portion of the natural language string; and
determining that a horizontal logical disjunction of a vertical logical conjunction of the security bitmask for the first branch and the security bitmask for the first user input data indicates authorization.
12. The non-transitory computer-readable storage medium of claim 9 , wherein:
obtaining the first user input data includes obtaining the first user input data from a user device; and
automatically outputting the results data includes outputting the results data to the user device.
13. The non-transitory computer-readable storage medium of claim 9 , wherein:
automatically indexing the enterprise data tokens includes:
including, in the index structure, data indicating an association between the enterprise data token and a first table stored in the database management system; and
including, in the index structure, data indicating an association between a second enterprise data token from the enterprise data tokens and a second table stored in the database management system;
automatically generating the semantic representation includes:
traversing the index structure to match a second portion of the natural language string to the second enterprise data token;
including the second enterprise data token in the semantic representation;
identifying a join path for joining data from the first table with data from the second table;
including data indicating the join path in the semantic representation; and
automatically converting the semantic representation includes:
including data indicating the join path in the structured query language query.
14. The non-transitory computer-readable storage medium of claim 13 , wherein:
obtaining the enterprise data includes:
obtaining relationship data indicating a relationship between the first table and the second table stored; and
storing the relationship data in the information retrieval system; and
identifying the join path includes using the relationship data.
15. The non-transitory computer-readable storage medium of claim 9 , wherein:
automatically indexing the enterprise data tokens includes:
including, in the index structure, data indicating an association between the enterprise data token and a column of a table stored in the database management system; and
automatically generating the semantic representation includes:
in response to determining, by a finite state machine of the information retrieval system, that the column is a measure column, including, in the semantic representation, data indicating an aggregation operation with respect to the measure column.
16. The non-transitory computer-readable storage medium of claim 9 , wherein:
automatically indexing the enterprise data tokens includes:
including, in the index structure, data indicating an association between the enterprise data token and a first column of a first table stored in the database management system;
automatically generating the semantic representation includes:
identifying candidate joint paths, wherein identifying the candidate join paths includes:
identifying a first candidate join path for joining data from the first column with data from a second column from a second table stored in the database management system; and
identifying a second candidate join path for joining data from the first column with data from a third column from the second table;
obtaining second user input data indicating a join path from the candidate join paths;
including data indicating the join path in the semantic representation; and
automatically converting the semantic representation includes:
including data indicating the join path in the structured query language query.
17. An apparatus of an information retrieval system comprising:
a non-transitory computer readable medium; and
a processor configured to execute instructions stored on the non-transitory computer readable medium to:
obtain, from a database management system, enterprise data;
automatically generate enterprise data tokens representing the enterprise data;
automatically index the enterprise data tokens in an index structure of the information retrieval system;
obtain first user input data including a natural language string expressing a request for data from the information retrieval system;
automatically generate a semantic representation of the natural language string, in a form that differs from a structured query language of the database management system, wherein to automatically generate the semantic representation the processor is configured to execute the instructions to:
traverse the index structure to match a portion of the natural language string to an enterprise data token from the enterprise data tokens; and
include the enterprise data token in the semantic representation;
automatically convert the semantic representation into a structured query language query expressing the request for data;
obtain, from the database management system, in response to the structured query language query, results data responsive to the request for data; and
automatically output, for presentation to a user, the results data.
18. The apparatus of claim 17 , wherein, to index the enterprise data tokens, the processor is configured to execute the instructions to:
include, in the index structure, a root node;
include, in the index structure, a first branch depending from the root node, the first branch representing a first symbol from the enterprise data token; and
include, in the index structure, a security bitmask for the enterprise data token, such that a security bitmask for the first branch at the root node is a hierarchical logical disjunction based on the security bitmask for the enterprise data token.
19. The apparatus of claim 18 , wherein:
to obtain the first user input data the processor is configured to execute the instructions to obtain a security bitmask for the first user input data; and
to automatically generate the semantic representation the processor is configured to execute the instructions to:
determine that the first symbol matches a symbol from the portion of the natural language string; and
determine that a horizontal logical disjunction of a vertical logical conjunction of the security bitmask for the first branch and the security bitmask for the first user input data indicates authorization.
20. The apparatus of claim 17 , wherein:
to automatically index the enterprise data tokens the processor is configured to execute the instructions to:
include, in the index structure, data indicating an association between the enterprise data token and a first table stored in the database management system; and
include, in the index structure, data indicating an association between a second enterprise data token from the enterprise data tokens and a second table stored in the database management system;
to automatically generate the semantic representation the processor is configured to execute the instructions to:
traverse the index structure to match a second portion of the natural language string to the second enterprise data token;
include the second enterprise data token in the semantic representation;
identify a join path for joining data from the first table with data from the second table;
include data indicating the join path in the semantic representation; and
to automatically convert the semantic representation the processor is configured to execute the instructions to:
include data indicating the join path in the structured query language query.