Systems and methods for batch processing of data records in an on-demand system
View Patent ↗Systems and methods are provided for batch processing of data records in an on-demand system. A marker is stored that identifies a node in an on-demand system and identifies a most recent data record processed in a relational database management system by the node. The marker is used to query the database management system to identify a batch of records for the node to process. The batch of records is processed by the node to update a client of data changes reflected by the batch of records.
1. An apparatus for batch processing of data records in an on-demand system, the apparatus comprising:
a processor; and
a non-transitory computer readable medium storing a plurality of instructions which,
when executed by the processor, cause the processor to carry out the steps of:
storing a first marker that identifies a first node in an on-demand system and identifies a most recent data record processed in a relational database management system by the first node;
storing a second marker that identifies a second node in the on-demand system and identifies a most recent data record processed in the relational database management system by the second node;
using the first and second markers to query the database management system to identify a first batch of records for the first node to process and a second batch of records for the second node to process; and
processing the first batch of records by the first node to update a first client of data changes reflected by the first batch of records and the second batch of records by the second node to update a second client of data changes reflected by the second batch of records.
2. The apparatus of claim 1 , wherein storing the first and second markers comprises storing the first and second markers in a transactions table associated with the relational database management system.
3. The apparatus of claim 1 , wherein each of the first and second markers further identifies a data record type.
4. The apparatus of claim 1 , wherein the first and second batches of records are each based on a maximum batch size.
5. A computer program product, comprising a non-transitory computer-readable medium carrying one or more sequences of instructions for batch processing of data records in an on-demand system, which instructions, when executed by one or more processors, cause the one or more processors to:
store a first marker that identifies a first node in an on-demand system and identifies a most recent data record processed in a relational database management system by the first node;
store a second marker that identifies a second node in the on-demand system and identifies a most recent data record processed in the relational database management system by the second node;
use the first and second markers to query the database management system to identify a first batch of records for the first node to process and a second batch of records for the second node to process; and
process the first batch of records by the first node to update a first client of data changes reflected by the first batch of records and the second batch of records by the second node to update a second client of data changes reflected by the second batch of records.
6. The computer program product of claim 5 , wherein storing the first and second markers comprises storing the first and second markers in a transactions table associated with the relational database management system.
7. The computer program product of claim 5 , wherein each of the first and second markers further identifies a data record type.
8. The computer program product of claim 5 , wherein the first and second batches of records are each based on a maximum batch size.
9. A method for batch processing of data records in an on-demand system, the method comprising:
storing a first marker that identifies a first node in an on-demand system and identifies a most recent data record processed in a relational database management system by the first node;
storing a second marker that identifies a second node in the on-demand system and identifies a most recent data record processed in the relational database management system by the second node;
using the first and second markers to query the database management system to identify a first batch of records for the first node to process and a second batch of records for the second node to process; and
processing the first batch of records by the first node to update a first client of data changes reflected by the first batch of records and the second batch of records by the second node to update a second client of data changes reflected by the second batch of records.
10. The method of claim 9 , wherein storing the first and second markers comprises storing the first and second markers in a transactions table associated with the relational database management system.
11. The method of claim 9 , wherein each of the first and second markers further identifies a data record type.
12. The method of claim 9 , wherein the first and second batches of records are each based on a maximum batch size.
13. A method for transmitting code for batch processing of data records in an on-demand system on a transmission medium, the method comprising:
transmitting code to store a first marker that identifies a first node in an on-demand system and identifies a most recent data record processed in a relational database management system by the first node;
transmitting code to store a second marker that identifies a second node in the on-demand system and identifies a most recent data record processed in the relational database management system by the second node;
transmitting code to use the first and second markers to query the database management system to identify a first batch of records for the first node to process and a second batch of records for the second node to process; and
transmitting code to process the first batch of records by the first node to update a first client of data changes reflected by the first batch of records and the second batch of records by the second node to update a second client of data changes reflected by the second batch of records.
14. The method for transmitting code of claim 13 , wherein storing the first and second markers comprises storing the first and second markers in a transactions table associated with the relational database management system.
15. The method for transmitting code of claim 13 , wherein each of the first and second markers further identifies a data record type.
16. The method for transmitting code of claim 13 , wherein the first and second batches of records are each based on a maximum batch size.