Data processing pipeline horizontal scaling
Techniques are disclosed for executing a data processing pipeline. The techniques may include receiving a job at a data pipeline queue, setting up one or more distributed processing environments, and allocating the job to one of the distributed processing environments. The techniques may further include receiving the allocated job at a job queue within the distributed processing environment, increasing a priority level of the job, and executing the job within the distributed processing environment. The techniques can further include providing a retry pipeline at the data processing pipeline, and re-executing the job at a stage following a failure of at least one of its components. The techniques may decrement the retry budget as the job is re-executed.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving, at an execution queue, a processing job associated with an operation of an autonomous vehicle;
determining that a scheduling component associated with a first distributed processing environment is associated with an overload attribute;
determining, by a controller, an overload state associated with the overload attribute;
generating, based at least in part on the overload state, a second distributed processing environment;
allocating, by the controller, the processing job to the second distributed processing environment;
determining, by the controller, that an actual completion time for the processing job is longer than an estimated completion time for the processing job; and
generating, by the controller, a third distributed processing environment.
2. The system of claim 1 , wherein the first distributed processing environment comprises:
the scheduling component; and
a plurality of computing resources allocated to the first distributed processing environment.
3. The system of claim 1 , wherein the overload attribute comprises at least one of:
a first number of processing jobs executing at the first distributed processing environment;
a second number of processing jobs within the execution queue;
a total processing job runtime;
an organizational attribution associated with the first distributed processing environment;
a processing job response time;
a processing job turnaround time; or
a wait time associated with the execution queue.
4. The system of claim 1 , wherein the first distributed processing environment is configured to operate independently from the second distributed processing environment.
5. The system of claim 1 , the operations further comprising:
allocating, by the controller, a second processing job to the first distributed processing environment or the second distributed processing environment based on at least one of:
a first load level of the first distributed processing environment and a second load level of the second distributed processing environment;
an execution constraint of the second processing job; or
a first hardware availability of the first distributed processing environment and a second hardware availability of the second distributed processing environment.
6. A method comprising:
receiving, at an execution queue, a processing job;
determining that a scheduling component of a first distributed processing environment is associated with an attribute;
determining, by a controller, an unavailable state associated with the attribute;
generating, based at least in part on the unavailable state, a second distributed processing environment;
allocating, by the controller, the processing job to the second distributed processing environment;
determining, by the controller, that an actual completion time for the processing job is longer than an estimated completion time for the processing job; and
generating, by the controller, a third distributed processing environment.
7. The method of claim 6 , wherein the first distributed processing environment comprises:
the scheduling component; and
a plurality of computing resources allocated to the first distributed processing environment.
8. The method of claim 6 , wherein the unavailable state is indicative of an overload state of the scheduling component.
9. The method of claim 6 , wherein the attribute comprises at least one of:
a first number of processing jobs executing at the first distributed processing environment;
a second number of processing jobs within the execution queue;
a total processing job runtime;
an organizational attribution associated with the first distributed processing environment;
a processing job response time;
a processing job turnaround time; or
a wait time associated with the execution queue.
10. The method of claim 6 , further comprising:
transmitting, based at least in part on the unavailable state of a component associated with the attribute at the first distributed processing environment, the component associated with the attribute from the second distributed processing environment to the first distributed processing environment.
11. The method of claim 6 , wherein the first distributed processing environment is configured to operate independently of the second distributed processing environment.
12. The method of claim 6 , further comprising:
receiving, at the execution queue, a second processing job;
determining that the second processing job is associated with a local processing component;
determining, by the controller, an unavailability state associated with the local processing component of the first distributed processing environment; and
receiving, at the execution queue and by the controller, a third processing job associated with an operation of a vehicle.
13. The method of claim 6 , further comprising:
allocating, by the controller, a second processing job to the first distributed processing environment or the second distributed processing environment based on at least one of:
a first load level of the first distributed processing environment and a second load level of the second distributed processing environment;
an execution constraint of the second processing job; or
a first hardware availability of the first distributed processing environment and a second hardware availability of the second distributed processing environment.
14. The method of claim 6 , further comprising:
receiving the estimated completion time for the processing job.
15. The method of claim 6 , further comprising:
receiving a threshold associated with a maximum amount of jobs executable at the second distributed processing environment; and
determining, by the controller, that an amount of jobs to be executed at the second distributed processing environment meets or exceeds the threshold.
16. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving, at an execution queue, a processing job;
determining that a scheduler component of a first distributed processing environment is associated with an attribute;
determining, by a controller, an unavailable state associated with the attribute;
generating, based at least in part on the unavailable state, a second distributed processing environment;
allocating, by the controller, the processing job to the second distributed processing environment;
determining, by the controller, that an amount of jobs to be executed at the second distributed processing environment meets or exceeds a threshold associated with a maximum amount of jobs executable at the second distributed processing environment; and
generating, by the controller, a third distributed processing environment.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the first distributed processing environment comprises:
the scheduler component; and
a plurality of computing resources allocated to the first distributed processing environment.
18. The one or more non-transitory computer-readable media of claim 16 , wherein the unavailable state is indicative of an overload state of the scheduler component.
19. The one or more non-transitory computer-readable media of claim 16 , wherein the attribute comprises at least one of:
a first number of processing jobs executing at the first distributed processing environment;
a second number of processing jobs within the execution queue;
a total processing job runtime;
an organizational attribution associated with the first distributed processing environment;
a processing job response time;
a processing job turnaround time; or
a wait time associated with the execution queue.
20. The one or more non-transitory computer-readable media of claim 16 , the operations further comprising:
transmitting, by the controller, a second processing job to the first distributed processing environment or the second distributed processing environment based on at least one of:
a first load level of the first distributed processing environment and a second load level of the second distributed processing environment;
an execution constraint of the second processing job; or
a first hardware availability of the first distributed processing environment and a second hardware availability of the second distributed processing environment.