Asynchronous document ingestion and enrichment system
Provided are asynchronous data ingestion and enrichment systems and methods. The systems comprise a plurality of components (e.g., ingestion components, enrichment components, and/or publishing components). Instead of passing data from one component to another, the data is sent to a messaging queue that formats and hold the data until the subsequent component is ready to receive it. Additionally, each component comprises a central microservice and a plurality of instances, the central microservice configured to communicate with each instance of the plurality of instances.
1. An asynchronous data ingestion and enrichment system comprising one or more processors and memory storing instructions executable by the one or more processors to:
instantiate a plurality of ingestion software processing modules, wherein each ingestion software processing module of the plurality of ingestion software processing modules comprises an ingestion central microservice configured to manage a plurality of ingestion instances;
instantiate a plurality of enrichment software processing modules, wherein each enrichment software processing module of the plurality of enrichment software processing modules comprises an enrichment central microservice configured to manage a plurality of enrichment instances;
instantiate a plurality of publishing software processing modules, wherein each publishing software processing module of the plurality of publishing software processing modules comprises a publishing central microservice configured to manage a plurality of publishing instances;
instruct each of the ingestion software processing modules to publish to one or more of a plurality of messaging queues, wherein at least one of the ingestion software processing modules are configured to apply a de-duplication algorithm, wherein applying the de-duplication algorithm comprises transmitting a message from the enrichment central microservice to an Application Processing Interface (“API”) to determine whether a document has been previously processed by the system and further comprises the enrichment central microservice calling back to at least one other component of the system to determine whether the document has been previously processed;
instruct each of the enrichment software processing modules to consume from one or more of the plurality of messaging queues and publish to one or more of the plurality of messaging queues, wherein at least one of the enrichment software processing modules is configured such that, in response to a first one of the plurality of enrichment instances of the enrichment software processing module ceasing operation, a second one of the enrichment instances of the plurality of enrichment instances resumes operation without duplication of work; and
instruct each of the publishing software processing modules to consume from one or more of the plurality of messaging queues.
2. The system of claim 1 , wherein managing the plurality of ingestion instances comprises at least one of instantiating or terminating of one or more ingestion instances of the plurality of ingestion instances.
3. The system of claim 1 , wherein managing the plurality of enrichment instances comprises at least one of instantiating or terminating of one or more enrichment instances of the plurality of enrichment instances.
4. The system of claim 1 , wherein managing the plurality of publishing instances comprises at least one of instantiating or terminating of one or more publishing instances of the plurality of publishing instances.
5. The system of claim 1 , wherein an administrative service is configured to instruct two or more enrichment software processing modules of the plurality of enrichment software processing modules to operate in parallel.
6. The system of claim 5 , wherein the instruction to operate in parallel comprises an instruction for the two or more enrichment software processing modules to each consume from a first messaging queue of the plurality of messaging queues.
7. The system of claim 1 , wherein an administrative service is configured to instruct two or more enrichment software processing modules of the plurality of enrichment software processing modules to operate in series.
8. The system of claim 7 , wherein the instruction to operate in series comprises an instruction for a first enrichment software processing module of the two or more enrichment software processing modules to publish to the first messaging queue of the plurality of messaging queues, and comprises an instruction for a second enrichment software processing module of the two or more enrichment software processing modules to consume from the first messaging queue of the plurality of messaging queues.
9. The system of claim 1 , wherein an administrative service is configured to track one or more metrics for one of more software processing modules selected from: the plurality of ingestion software processing modules, the plurality of enrichment software processing modules, and the plurality of publishing software processing modules.
10. The system of claim 9 , wherein the one or more metrics comprise one or more of: a data throughput metric, a number of documents processed, a data traffic metric, a workload metric, and a document processing rate metric.
11. The system of claim 9 , wherein the administrative service is configured to instantiate or terminate one or more instances of one or more of the one or more software processing modules based on one or more of the tracked metrics.
12. The system of claim 1 , wherein an administrative service is configured to ensure that a dataset is consumed by only a single software processing module.
13. A method for asynchronous data ingestion and enrichment system, the method performed at a system comprising one or more processors and memory storing instructions for executing the method, the method comprising:
instantiating a plurality of ingestion software processing modules, wherein each ingestion software processing module of the plurality of ingestion software processing modules comprises an ingestion central microservice configured to manage a plurality of ingestion instances;
instantiating a plurality of enrichment software processing modules, wherein each enrichment software processing module of the plurality of enrichment software processing modules comprises an enrichment central microservice configured to manage a plurality of enrichment instances;
instantiating a plurality of publishing software processing modules, wherein each publishing software processing module of the plurality of publishing software processing modules comprises a publishing central microservice configured to manage a plurality of publishing instances; and
instructing each of the ingestion software processing modules to publish to one or more of a plurality of messaging queues, wherein at least one of the ingestion software processing modules are configured to apply a de-duplication algorithm, wherein applying the de-duplication algorithm comprises transmitting a message from the enrichment central microservice to an Application Processing Interface (“API”) to determine whether a document has been previously processed by the system and further comprises the enrichment central microservice calling back to at least one other component of the system to determine whether the document has been previously processed;
instructing each of the enrichment software processing modules to consume from one or more of the plurality of messaging queues and publish to one or more of the plurality of messaging queues, wherein at least one of the enrichment software processing modules is configured such that, in response to a first one of the plurality of enrichment instances of the enrichment software processing module ceasing operation, a second one of the enrichment instances of the plurality of enrichment instances resumes operation without duplication of work; and
instructing each of the publishing software processing modules to consume from one or more of the plurality of messaging queues.
14. The method of claim 13 , wherein managing the plurality of ingestion instances comprises at least one of instantiating or terminating of one or more ingestion instances of the plurality of ingestion instances.
15. The method of claim 13 , wherein managing the plurality of enrichment instances comprises at least one of instantiating or terminating of one or more enrichment instances of the plurality of enrichment instances.
16. The method of claim 13 , wherein managing the plurality of publishing instances comprises at least one of instantiating or terminating of one or more publishing instances of the plurality of publishing instances.
17. The method of claim 13 , wherein the method comprises instructing, by an administrative service, two or more enrichment software processing modules of the plurality of enrichment software processing modules to operate in parallel.
18. The method of claim 13 , wherein the method comprises instructing, by an administrative service, two or more enrichment software processing modules of the plurality of enrichment software processing modules to operate in series.
19. A non-transitory computer-readable storage medium for asynchronous data ingestion and enrichment system, the non-transitory computer-readable storage medium storing instructions configured to be executed by one or more processors of a system comprising one or more processors configured to cause the system to:
instantiate a plurality of ingestion software processing modules, wherein each ingestion software processing module of the plurality of ingestion software processing modules comprises an ingestion central microservice configured to manage a plurality of ingestion instances;
instantiate a plurality of enrichment software processing modules, wherein each enrichment software processing module of the plurality of enrichment software processing modules comprises an enrichment central microservice configured to manage a plurality of enrichment instances;
instantiate a plurality of publishing software processing modules, wherein each publishing software processing module of the plurality of publishing software processing modules comprises a publishing central microservice configured to manage a plurality of publishing instances; and
instruct each of the ingestion software processing modules to publish to one or more of a plurality of messaging queues, wherein at least one of the ingestion software processing modules are configured to apply a de-duplication algorithm, wherein applying the de-duplication algorithm comprises transmitting a message from the enrichment central microservice to an Application Processing Interface (“API”) to determine whether a document has been previously processed by the system and further comprises the enrichment central microservice calling back to at least one other component of the system to determine whether the document has been previously processed;
instruct each of the enrichment software processing modules to consume from one or more of the plurality of messaging queues and publish to one or more of the plurality of messaging queues, wherein at least one of the enrichment software processing modules is configured such that, in response to a first one of the plurality of enrichment instances of the enrichment software processing module ceasing operation, a second one of the enrichment instances of the plurality of enrichment instances resumes operation without duplication of work; and
instruct each of the publishing software processing modules to consume from one or more of the plurality of messaging queues.