Plug-and-ingest framework for question answering systems
A system for question answering (QA) documents data ingestion decides to ingest the documents data through a first plurality of sub-pipelines including a first sub-pipeline having a first set of engines and a second sub-pipeline having a second set of engines being independent from the first set of engines. The system determines a subset of the documents data and decides to ingest the subset through a second plurality of sub-pipelines including a third sub-pipeline having a third set of engines and a fourth sub-pipeline having a fourth set of engines being independent from the third set of engines. A set of engines of the second plurality of sub-pipelines and a set of engines of the first plurality of sub-pipelines are in a common class. The system selects output data from the second plurality of sub-pipelines over corresponding output data from the first plurality of sub-pipelines and generates a knowledge base.
1. A method for documents data ingestion, the method comprising:
ingesting a plurality of input documents by a first full sub-pipeline of an ingestion pipeline to extract first relationship data and first entities data for building a knowledge base;
applying, by the ingestion pipeline, a filtering rule to the plurality of input documents to generate a filtered documents subset of the plurality of input documents;
ingesting the filtered documents subset of the plurality of input documents by a first exclusive sub-pipeline of the ingestion pipeline to extract second relationship data and second entities data;
inputting the first relationship data and the second relationship data into a knowledge graph selector;
selecting, by the knowledge graph selector, first data records from the first relationship data and the second relationship data based on a first set of selection rules for building a knowledge graph of the knowledge base;
inputting the first entities data and the second entities data into a search index selector; and
selecting, by the search index selector, second data records from the first entities data and the second entities data based on a second set of selection rules for building an index collection of the knowledge base.
2. The method of claim 1 , further comprising:
ingesting the plurality of input documents by a second full sub-pipeline of the ingestion pipeline to extract first concept data for building the knowledge base;
inputting the first concept data into the search index selector; and
selecting, by the search index selector, the second data records from the first entities data, the second entities data, and the first concept data based on the second set of selection rules for building the index collection of the knowledge base.
3. The method of claim 1 , wherein ingesting the plurality of input documents by the first full sub-pipeline and ingesting the filtered documents subset of the plurality of input documents by the first exclusive sub-pipeline are performed concurrently and in parallel.
4. The method of claim 1 , wherein the filtered documents subset of the plurality of input documents comprises of a number of documents below a predetermined threshold of the plurality of input documents for processing by the first exclusive sub-pipeline of the ingestion pipeline.
5. The method of claim 2 , further comprising:
ingesting the plurality of input documents by the second full sub-pipeline of the ingestion pipeline to extract third relationship data for building the knowledge base;
inputting the third relationship data into the search index selector; and
selecting, by the search index selector, the second data records from the first entities data, the second entities data, the first concept data, and the third relationship data based on the second set of selection rules for building the index collection of the knowledge base.
6. The method of claim 5 , further comprising:
ingesting the filtered documents subset of the plurality of input documents by a second exclusive sub-pipeline of the ingestion pipeline to extract fourth relationship data;
inputting the fourth relationship data into the search index selector; and
selecting, by the search index selector, the second data records from the first entities data, the second entities data, the first concept data, the third relationship data, and the fourth relationship data based on the second set of selection rules for building the index collection of the knowledge base.
7. The method of claim 1 , further comprising extending an abstract sub-pipeline to create both the first full sub-pipeline and the first exclusive sub-pipeline of the ingestion pipeline.
8. The method of claim 1 , further comprising modifying a sub-pipeline of the ingestion pipeline without disrupting the documents data ingestion of other sub-pipelines of the ingestion pipeline.
9. The method of claim 8 , wherein modifying the sub-pipeline of the ingestion pipeline comprises modifying a sub-pipeline model of the sub-pipeline.
10. The method of claim 8 , wherein modifying the sub-pipeline comprises modifying, adding, or removing an engine of the sub-pipeline.
11. The method of claim 8 , wherein modifying the sub-pipeline comprises adding or removing the sub-pipeline to the ingestion pipeline.
12. A system for documents data ingestion, the system comprising:
a memory having instructions therein; and
at least one processor in communication with the memory, wherein the at least one processor is configured to execute the instructions to:
ingest a plurality of input documents by a first full sub-pipeline of an ingestion pipeline to extract first relationship data and first entities data for building a knowledge base;
apply, by the ingestion pipeline, a filtering rule to the plurality of input documents to generate a filtered documents subset of the plurality of input documents;
ingest the filtered documents subset of the plurality of input documents by a first exclusive sub-pipeline of the ingestion pipeline to extract second relationship data and second entities data;
input the first relationship data and the second relationship data into a knowledge graph selector;
select, by the knowledge graph selector, first data records from the first relationship data and the second relationship data based on a set of selection rules for building a knowledge graph of the knowledge base;
input the first entities data and the second entities data into a search index selector; and
select, by the search index selector, second data records from the first entities data and the second entities data based on a second set of selection rules for building an index collection of the knowledge base.
13. The system of claim 12 , wherein the at least one processor is further configured to execute the instructions to:
ingest the plurality of input documents by a second full sub-pipeline of the ingestion pipeline to extract first concept data for building the knowledge base;
input the first concept data into the search index selector; and
select, by the search index selector, the second data records from the first entities data, the second entities data, and the first concept data based on the second set of selection rules for building the index collection of the knowledge base.
14. The system of claim 13 , wherein the at least one processor is further configured to execute the instructions to:
ingest the plurality of input documents by the second full sub-pipeline of the ingestion pipeline to extract third relationship data for building the knowledge base;
input the third relationship data into the search index selector; and
select, by the search index selector, the second data records from the first entities data, the second entities data, the first concept data, and the third relationship data based on the second set of selection rules for building the index collection of the knowledge base.
15. The system of claim 12 , wherein the at least one processor is further configured to execute the instructions to generate the filtering rule based on undesirable data from the knowledge base.
16. The system of claim 12 , wherein the at least one processor is further configured to execute the instructions to extend an abstract sub-pipeline to create both the full sub-pipeline and the exclusive sub-pipeline of the ingestion pipeline.
17. The system of claim 12 , wherein the at least one processor is further configured to execute the instructions to modify a sub-pipeline of the ingestion pipeline without disrupting the documents data ingestion of other sub-pipelines of the ingestion pipeline.
18. A computer program product for documents data ingestion, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by at least one processor to cause the at least one processor to:
ingest a plurality of input documents by a first full sub-pipeline of an ingestion pipeline to extract first relationship data and first entities data for building a knowledge base;
apply, by the ingestion pipeline, a filtering rule to the plurality of input documents to generate a filtered documents subset of the plurality of input documents;
ingest the filtered documents subset of the plurality of input documents by a first exclusive sub-pipeline of the ingestion pipeline to extract second relationship data and second entities data;
input the first relationship data and the second relationship data into a knowledge graph selector;
select, by the knowledge graph selector, first data records from the first relationship data and the second relationship data based on a set of selection rules for building a knowledge graph of the knowledge base;
input the first entities data and the second entities data into a search index selector; and
select, by the search index selector, second data records from the first entities data and the second entities data based on a second set of selection rules for building an index collection of the knowledge base.
19. The computer program product of claim 18 , wherein the program instructions executable by at least one processor further comprises instructions to:
ingest the plurality of input documents by a second full sub-pipeline of the ingestion pipeline to extract first concept data for building the knowledge base;
input the first concept data into the search index selector; and
select, by the search index selector, the second data records from the first entities data, the second entities data, and the first concept data based on the second set of selection rules for building the index collection of the knowledge base.
20. The computer program product of claim 18 , wherein the program instructions executable by at least one processor further comprises instructions to generate the filtering rule based on undesirable data from the knowledge base.