Neural processing device and method for job scheduling thereof
A neural processing device and a method for job scheduling are provided. The neural processing device configured to receive, by an address space ID (ASID) manager, first and second requests from at least one context, respectively, and determine whether ASIDs are allocated, store jobs of contexts to which the ASIDs have not been allocated from the ASID manager in entities, schedule, by a job scheduler, an execution order of the jobs stored in the entities and cause the ASID manager to allocate the ASIDs to the contexts to which the ASIDs have not been allocated among the at least one context, and sequentially receive, by a command queue, jobs of contexts to which the ASIDs have been allocated, store the jobs as standby jobs, and sequentially execute the standby jobs.
1. A neural processing device including at least one neural processor, a shared memory shared by the at least one neural processor, and a global interconnection configured to exchange data between the at least one neural processor and the shared memory,
the neural processing device configured to:
receive, by an address space ID (ASID) manager, first and second requests from at least one context, respectively, and determine whether ASIDs are allocated;
store jobs of contexts to which the ASIDs have not been allocated from the ASID manager in entities;
schedule, by a job scheduler, an execution order of the jobs stored in the entities and cause the ASID manager to allocate the ASIDs to the contexts to which the ASIDs have not been allocated among the at least one context; and
sequentially receive, by a command queue, jobs of contexts to which the ASIDs have been allocated, store the jobs as standby jobs, and sequentially execute the standby jobs, wherein the standby jobs comprise a first job transmitted directly without going through the job scheduler and a second job transmitted through an allocation procedure of the ASIDs by the job scheduler.
2. The neural processing device of claim 1 , wherein the at least one context comprises first and second contexts,
the first context is allocated a first ASID from the ASID manager,
the second context is not allocated any of the ASIDs from the ASID manager and then is allocated one of the ASIDs by the job scheduler, and
the first and second contexts perform the first and second jobs, respectively.
3. The neural processing device of claim 2 , wherein the at least one context further comprises a third context, and
the third context is allocated a second ASID from the ASID manager.
4. The neural processing device of claim 2 , wherein the at least one context further comprises a fourth context,
the entities comprise first and second entities,
the first entity receives the second job from the second context, and
the second entity receives a third job from the fourth context.
5. The neural processing device of claim 4 , wherein the job scheduler schedules jobs stored in the first and second entities and thereby allocates the jobs to the command queue.
6. The neural processing device of claim 5 , wherein the ASID manager provides a pass-through route and a non-pass-through route,
the pass-through route is a path to which the ASID manager directly allocates the first and second ASIDs, and
the non-pass-through route is a path to which the ASID manager does not directly allocate an ASID and then the job scheduler allocates an ASID.
7. The neural processing device of claim 6 , wherein the job scheduler requests allocation of one or more currently unbound ASID to the ASID manager.
8. The neural processing device of claim 2 , wherein the second context wakes up the job scheduler by pushing the second job,
the job scheduler transfers the second job to the command queue by emitting the second job, and
the neural processing device executes the second job.
9. The neural processing device of claim 8 , wherein there exists a first scheduling overhead between a time point at which the second context pushes the second job and a time point before the job scheduler emits the second job.
10. The neural processing device of claim 9 , wherein there exists a second scheduling overhead between a time point when the job scheduler emits the second job and a time point before the neural processing device executes the second job.
11. The neural processing device of claim 8 , wherein the first context transfers the first job to the neural processing device without a scheduling overhead, and
the neural processing device executes the first job.
12. The neural processing device of claim 1 , wherein the at least one context comprises first and second contexts,
the first context is allocated a first ASID from the ASID manager, and
the ASID manager unbinds the first ASID when the first job is executed.
13. The neural processing device of claim 12 , wherein the second context is not allocated one of the ASIDs from the ASID manager, and then is allocated one of unbound ASIDs from the ASID manager according to a request of the job scheduler.
14. The neural processing device of claim 13 , wherein the ASID manager allocates an LRU (least recently used) ASID that has not been used for a longest time among the unbound ASIDs to the second context.