Apparatus, systems, and methods for batch and realtime data processing
View Patent ↗A traditional data processing system is configured to process input data either in batch or in real-time. On one hand, a batch data processing system is limiting because the batch data processing often cannot take into account any data received during the batch data processing. On the other hand, a real-time data processing system is limiting because the real-time system often cannot scale. The real-time data processing system is often limited to dealing with primitive data types and/or a small amount of data. Therefore, it is desirable to address the limitations of the batch data processing system and the real-time data processing system by combining the benefits of the batch data processing system and the real-time data processing system into a single data processing system.
1. A computing system for generating a summary data of a set of data, the computing system comprising:
one or more processors configured to run one or more modules stored in non-tangible computer readable medium, wherein the one or more modules are operable to:
receive a first set of data and a second set of data, wherein the first set of data comprises a larger number of data items compared to the second set of data;
process the first set of data to format the first set of data into a first structured set of data;
generate a first summary data using the first structured set of data by operating rules for summarizing the first structured set of data, and store the first summary data in a data store;
process the second set of data to format the second set of data into a second structured set of data;
generate a second summary data based on the first structured set of data and the second structured set of data by operating rules for summarizing the first structured set of data and the second structured set of data;
determine a difference between the first summary data and the second summary data; and
update the data store based on the difference between the first summary data and the second summary data.
2. The computing system of claim 1 , wherein the second set of data comprises real-time data submissions, and the one or more modules are operable to process the second set of data to format the second set of data into the second structured set of data in response to receiving the second set of data.
3. The computing system of claim 1 , wherein the one or more modules are operable to process the first set of data to format the first set of data into the first structured set of data at a first time interval, which is substantially longer than a second time interval at which the second set of data is formatted into the second structured set of data.
4. The computing system of claim 1 , wherein each of the first summary data and the second summary data comprises an entity identifier and a value associated with the entity identifier, and wherein the one or more modules are operable to determine the difference between the first summary data and the second summary data by:
determining that the first summary data and the second summary data include an identical entity identifier, and
comparing values associated with the identical entity identifiers in the first summary data and the second summary data.
5. The computing system of claim 4 , wherein the one or more modules are further operable to provide the difference between the first summary data and the second summary data to other authorized computing systems.
6. The computing system of claim 5 , wherein the one or more modules are further operable to provide the difference to other authorized computing systems via an application programming interface.
7. The computing system of claim 5 , wherein the one or more modules are further operable to provide the difference to other authorized computing systems as a file.
8. The computing system of claim 1 , wherein the one or more modules are operable to:
combine at least the first set of data and the second set of data to generate a third set of data;
process the third set of data to format the third set of data into a third structured set of data based on new rules for formatting a set of data; and
generate a third summary data using the third structured set of data.
9. The computing system of claim 8 , wherein the first set of data and the third set of data each includes a first data element, and wherein the first data element is associated with a first entity in the first summary data identified by the first entity identifier, wherein the first data element is associated with a second entity in the third summary data, and wherein the one or more modules are further operable to associate the first entity identifier to the second entity in the third summary data so that the first data element maintains its association with the first entity identifier in the third summary data.
10. The computing system of claim 1 , wherein the first structured set of data comprises a grouping of data items based on an entity identifier associated with the data items.
11. The computing system of claim 1 , wherein the computing system comprises at least one server in a data center.
12. The computing system of claim 1 , wherein the data store comprises a plurality of database systems, each of which is associated with a view, and wherein the one or more modules are operable to select one of the plurality of database systems in response to a query based on the view associated with the query.
13. The computing system of claim 1 , wherein the one or more modules are operable to identify a third set of data received after the generation of the second summary data;
generate a third summary data based on the third set of data, the first structured set of data, and the second structured set of data by operating rules for summarizing the first structured set of data, the second structured set of data, and the third summary data;
determine a difference between the second summary data and the third summary data; and
update the data store based on the difference between the second summary data and the third summary data.
14. A method for generating a summary data of a set of data, the method comprising:
receiving, at an input module operating on a processor of a computing system, a first set of data and a second set of data, wherein the first set of data comprises a larger number of data items compared to the second set of data;
processing, at a first input processing module of the computing system, the first set of data to format the first set of data into a first structured set of data;
generating, at a first summary generation module of the computing system, a first summary data using the first structured set of data by operating rules for summarizing the first structured set of data;
maintaining the first summary data in a data store in the computing system;
processing, at a second input processing module of the computing system, the second set of data to format the second set of data into a second structured set of data;
generating, at a second summary generation module of the computing system, a second summary data using the first structured set of data and the second structured set of data by operating rules for summarizing the first structured set of data and the second structured set of data;
determining, at a difference generation module of the computing system, a difference between the first summary data and the second summary data; and
updating, by the computing system, the data store based on the difference between the first summary data and the second summary data.
15. The method of claim 14 , wherein receiving the second set of data comprises receiving real-time data submissions, and wherein processing the second set of data to format the second set of data into the second structured set of data comprises processing the second set of data in response to receiving the second set of data.
16. The method of claim 14 , wherein each of the first summary data and the second summary data comprises an entity identifier and a value associated with the entity identifier, and wherein determining the difference between the first summary data and the second summary data comprises:
determining that the first summary data and the second summary data include an identical entity identifier, and
comparing values associated with the identical entity identifiers in the first summary data and the second summary data.
17. The method of claim 14 , further comprising:
combining at least the first set of data and the second set of data to generate a third set of data,
processing the third set of data to format the third set of data into a third structured set of data based on new rules for formatting a set of data; and
generating a third summary data using the third structured set of data.
18. The method of claim 14 , wherein a first data element is a part of the first set of data and the third set of data, and is associated with a first entity, identified by the first entity identifier, in the first summary data, wherein the first data element is associated with a second entity in the third summary data, and wherein the method further comprises associating the first entity identifier to the second entity in the third summary data so that the first data element maintains its association with the first entity identifier in the third summary data.
19. The method of claim 14 , wherein the computing system comprises at least one server in a data center.
20. A computer program product, tangibly embodied in a non-transitory computer-readable storage medium, the computer program product including instructions operable to cause a data processing system to:
receive a first set of data and a second set of data, wherein the first set of data comprises a larger number of data items compared to the second set of data;
process the first set of data to format the first set of data into a first structured set of data;
generate a first summary data using the first structured set of data by operating rules for summarizing the first structured set of data, and store the first summary data in a data store;
process the second set of data to format the second set of data into a second structured set of data;
generate a second summary data using the first structured set of data and the second structured set of data by operating rules for summarizing the first structured set of data and the second structured set of data;
determine a difference between the first summary data and the second summary data; and
update the data store based on the difference between the first summary data and the second summary data.
21. The computer program product of claim 20 , wherein the second set of data comprises real-time data submissions, and wherein the instructions are operable to cause the data processing system to process the second set of data to format the second set of data into the second structured set of data in response to receiving the second set of data.
22. The computer program product of claim 20 , wherein each of the first summary data and the second summary data comprises an entity identifier and a value associated with the entity identifier, and wherein the instructions are operable to cause the data processing system to determine the difference between the first summary data and the second summary data by:
determining that the first summary data and the second summary data include an identical entity identifier, and
comparing values associated with the identical entity identifiers in the first summary data and the second summary data.
23. The computer program product of claim 20 , wherein the instructions are operable to cause the data processing system to:
combine at least the first set of data and the second set of data to generate a third set of data,
process the third set of data to format the third set of data into a third structured set of data based on new rules for formatting a set of data; and
generate a third summary data using the third structured set of data.