Data processing systems and methods
Example data processing systems and methods are described. In one implementation, a system accesses a corpus of data and analyzes the data contained in the corpus of data to identify multiple documents. The system generates vector indexes for the multiple documents such that the vector indexes allow a computing system to quickly access the plurality of documents and identify an answer to a question associated with the corpus of data.
1. A method for processing messages of answering a query, the method comprising:
receiving, by a computing system, the query from a user;
determining, by the computing system, an identity of the user;
generating, by the computing system, a query vector for the query;
accessing, by the computing system, a knowledge base associated with the query, wherein the knowledge base comprises a plurality of vector indexes, and wherein each vector index comprises a plurality of vectors associated with a respective data element;
determining a respective score for each of the plurality of data elements by comparing the query vector to the plurality of vectors of the plurality of vector indexes;
in response to none of the respective scores for the plurality of data elements being above a threshold value, communicating, by the computing system, at least two data elements of the plurality of data elements having the highest respective score to the user;
and
in response to at least one of the respective scores for the plurality of data elements being above a threshold value, communicating, by the computing system, a first data element having the highest respective score to the user.
2. The method of claim 1 , wherein the identity of the user is determined by using a cookie that identifies a computing device being used by the user.
3. The method of claim 1 , comprising identifying a category for the knowledge base based on an intent associated with the query.
4. The method of claim 1 , comprising determining whether an answer to the query needs disambiguation.
5. The method of claim 4 , comprising prompting the user to disambiguate the answer to the query by selecting a particular category for the knowledge base responsive to determining that the answer to the query needs disambiguation.
6. The method of claim 1 , wherein receiving, by the computing system, the query comprises using a conversational bot trained to respond to user queries.
7. The method of claim 1 , comprising filtering the plurality of data elements.
8. The method of claim 7 , wherein the filtering of the plurality of data elements is based on metadata of the plurality of data elements.
9. The method of claim 1 , wherein comparing the query vector to the plurality of vectors of the plurality of vector indexes comprises using a nearest neighbor technique.
10. The method of claim 1 , wherein determining the respective score for each of the plurality of data elements comprises applying a respective scoring function.
11. A run time processing system comprising:
a communication manager configured to access a knowledge base containing a plurality of vector indexes, wherein each vector index comprises a plurality of vectors associated with a respective data element of a plurality of data elements;
a processor configured to receive a query associated with the knowledge base, wherein the query is received from a user and
identifies the user;
a search module configured to:
generate a query vector for the query;
determine a respective score for each of the plurality of data elements by comparing the query vector to the plurality of vectors of the plurality of vector indexes;
in response to none of the respective scores for the plurality of data elements being above a threshold value, identify at least two data elements of the plurality of data elements having the highest respective score, wherein an answer to the query is generated based on the at least two data elements; and
in response to at least one of the respective scores for the plurality of data elements being above a threshold value, identify a first data element having the highest respective score, wherein the answer to the query is generated based on the first data element; and
a response manager communicates the answer to the user.
12. The run time processing system of claim 11 , wherein identifying the user comprises using a cookie to identify a computing device being used by the user.
13. The run time processing system of claim 11 , comprising an intent classification module configured to classify an intent associated with the query.
14. The run time processing system of claim 11 , comprising a disambiguation module configured to determine whether the answer needs disambiguation.
15. The run time processing system of claim 14 , wherein the communication manager is configured to prompt the user to disambiguate the answer responsive to determining that the answer needs disambiguation.
16. The run time processing system of claim 11 , comprising a ranking module configured to rank the plurality of data elements based on the respective scores.
17. The run time processing system of claim 16 , comprising a filtering module configured to filter the plurality of data elements.
18. The run time processing system of claim 17 , wherein the filtering module is configured to filter the plurality of data elements based on metadata of the plurality of data elements.
19. The run time processing system of claim 11 , wherein the answer comprises a respective portion of each of the at least two data elements or the first data element.
20. The run time processing system of claim 19 , wherein the respective portion is identified by matching a specific word in the query that appears in the respective portion.