System and method for dynamic data ingestion

1. A dynamic ingestion method comprising:

detecting presence of data sets in an input queue, each data set includes aggregate holdings belonging to a group, the aggregate holdings bundled by the input queue based on the group;

calculating a first amount of partitioned data processors required to process the detected data sets and a second amount of partitions required to process the detected data sets;

fetching the data sets from each group utilizing a parallel computation engine, the parallel computation engine including multiple partitioned processors and a business logic processor pool decoupled from the multiple partition processors that is utilizable by each of the multiple partitioned processors, wherein each group is assigned to a partition of one of the multiple partitioned processors for processing;

processing the data sets for each group utilizing a group level process of the multiple partitioned processors to obtain resultant data sets, the resultant data sets including at least one report comprising the aggregate holdings identified as breaching at least one stipulated threshold;

dynamically adjusting a quantity of business logic processors in the business logic processor pool based on available resources;

transmitting the resultant data sets for each group to the business logic processor pool in order to process the data sets in accordance with existing rules in order to generate a final data set; and

transmitting the final data set for each group to an output queue, the output queue decoupling the parallel computation engine from a data persistence technology by staging the final data set for the data persistence technology,

wherein the existing rules include one or more regulatory protocols from a regulatory agency, and

wherein the existing rules are stored in plug-in components that are pluggable into the parallel computation engine, each plug-in component being dynamically updated, added, or removed independently from the parallel computation engine.

2. The method of claim 1 , further comprising bundling received data into the datasets at the input queue.

3. The method of claim 1 , further comprising:

dynamically adjusting a number of partitions in at least one partitioned processor; and

dynamically adjusting a quantity of datasets fetched by each partitioned processor.

4. The method of claim 1 , further comprising sharing the business logic processor pool across multiple partitions.

5. The method of claim 1 , further comprising feeding a next set of data to the business logic processor pool when the business logic processor pool generates the final data set.

6. The method of claim 1 , further comprising dynamically adjusting a quantity of business logic processors in the business logic processor pool based on throughput.

7. The method of claim 1 , further comprising staging the final data set in the output queue prior to persistence.

8. The method of claim 1 , wherein each group is a company and the data corresponds to aggregate positions.

9. A dynamic ingestion system comprising:

an input queue storing data sets, each data set includes aggregate holdings belonging to a group, the aggregate holdings bundled by the input queue based on the group;

a parallel computation engine

calculating a first amount of partitioned data processors required to process the detected data sets and a second amount of partitions required to process the detected data sets;

fetching the data sets from each group of data in the input queue, the parallel computation engine including multiple partitioned processors and a business logic processor pool decoupled from the multiple partitioned processors that is utilizable by each of the multiple partitioned processors; and

dynamically adjusting a quantity of business logic processors in the business logic processor pool based on available resources,

wherein each group is assigned to a partition of one of the multiple partitioned processors for processing and each partition processes the data sets for each group utilizing a group level process to obtain resultant data sets, the resultant data sets including at least one report comprising the aggregate holdings identified as breaching at least one stipulated threshold;

the business logic processor pool receiving the resultant data sets and processing the resultant data sets in accordance with existing rules in order to generate a final data set; and

an output queue for receiving the final data sets, the output queue decoupling the parallel computation engine from a data persistence technology by staging the final data sets for the data persistence technology,

wherein the existing rules include one or more regulatory protocols from a regulatory agency, and

wherein the existing rules are stored in plug-in components that are pluggable into the parallel computation engine, each plug-in component being dynamically updated, added, or removed independently from the parallel computation engine.

10. The system of claim 9 , wherein the input queue bundles received data into the datasets.

11. The system of claim 9 , wherein the parallel computation engine dynamically adjusts a number of partitions in at least one partitioned processor, and dynamically adjusts a quantity of datasets fetched by each partitioned processor.

12. The system of claim 9 , wherein the business logic processor pool is shared across multiple partitions.

13. The system of claim 9 , wherein the parallel computation engine feeds a next set of data to the business logic processor pool when the business logic processor pool generates the final data set.

14. The system of claim 9 , wherein the parallel computation engine dynamically adjusts a quantity of business logic processors in the business logic processor pool based on throughput.

15. The system of claim 9 , wherein the output queue stages the final data set prior to persistence.

16. The system of claim 9 , wherein each group is a company and the data corresponds to aggregate positions.

17. The method of claim 1 , wherein the group level process for processing the data sets for each group comprises:

generating multiple parallel processes within each processor to operate on the data sets for each group.

18. The system of claim 9 , wherein the group level process for processing the data sets for each group comprises:

generating multiple parallel processes within each processor to operate on the data sets for each group.

19. The method of claim 4 , wherein the shared business logic pool is operable to control the parallel processing of the multiple partitioned processors.

20. The system of claim 12 , wherein the shared business logic pool is operable to control the parallel processing of the multiple partitioned processors.