Combining batch and queueable technologies in a salesforce platform for large volume parallel processing
Disclosed embodiments include a method for combining a BATCH application programming interface (API) and a QUEUEABLE API on a SALESFORCE platform to process records. The method can include accessing a programming environment associated with an add-on application for a SALESFORCE platform, obtaining records via the add-on application to be processed on the SALESFORCE platform in accordance with a job type, calling a BATCH API to process the records by batching the records to create job records, and calling a QUEUEABLE API to process the job records in QUEUEABLES including parallel QUEUEABLES such that at least some job records included in the parallel QUEUEABLES are processed in parallel in accordance with the job type.
1. A method for achieving improved throughput and finer control in processing a plurality of records by asynchronously combining batching and a plurality of queueables performed in accordance with a cloud-based computing architecture, the method comprising:
accessing a programming environment associated with an application for an application development platform of the cloud-based computing architecture;
obtaining the plurality of records via the application to be processed on the application development platform in accordance with a job type;
executing a first asynchronous process by calling a batching application programing interface (API) on the application development platform for limited asynchronous batching of the plurality of records to create a plurality of job records by generating a job table to process the plurality of records, the job table being populated with a record entry and unique identifier for each of the plurality of records;
calling a queueing application programing interface (API) to asynchronously process the plurality of job records from the job table in the plurality of queueables by determining a scope size for each of the plurality of queueables, the scope size including a number of the plurality of job records, and allocating the job records to the plurality of queueables in accordance with the scope size for each of the plurality of queueables, wherein the plurality of queueables includes a plurality of parallel queueables such that at least some job records included in the plurality of parallel queueables are processed in parallel in accordance with the job type; and
increasing a number of the plurality of queueables and executing a second asynchronous process by chaining at least one of the plurality of queueables to another queueable, the at least one queueable being initiated to run in parallel processing while the another queueable is running, wherein an identifier associated with the at least one queueable indicates a last time a job record was modified and is written in one or more of the plurality of job records, and wherein another identifier associated with the another queuable determines how the job record will be processed and is written to another one or more of the plurality of job records.
2. The method of claim 1 , wherein the plurality of queueables are arranged in a chain such that a first plurality of the plurality of queueables start processing the one or more job records in response to a second plurality of the plurality of queueables completing processing of the other one or more job records.
3. The method of claim 2 , wherein the first plurality of the plurality of queueables comprises the plurality of parallel queueables and the second plurality comprises another plurality of parallel queueables such that at least some records included in the second plurality of the plurality of queueables are processed in parallel.
4. The method of claim 1 , wherein the unique identifier is a record identifier that uniquely identifies the job record from other job records referred to in the job table.
5. The method of claim 4 , wherein each record entry comprises values for parameters including any of:
a call identifier representing a call that initiated the batch;
a queueable identifier indicating a queueable of the plurality of queueables that was given the job record to process;
time information indicating the last time the job record was modified;
a job type identifier indicating how the job record should be processed; and
status information indicating a processing status of the job record.
6. The method of claim 1 , wherein each record entry comprises status information including a status of a job record, the method further comprising:
polling the job table for the status in a record entry of each job record;
initiating reprocessing of the job record in accordance with the status of the job record; and
reprocessing any job record in accordance with the job type.