Interoperable neural network operation scheduler
A Neural Network (NN) scheduler and techniques to implement features of different possible NN schedulers are disclosed. In a first example, an NN scheduler that accepts NN models in an interoperable format and performs optimizations on this interoperable format as part of converting it to a run-time format is provided. In a second example, an NN scheduler analyzes operations and annotations associated with those operations to determine scheduling options based on hardware availability, data availability, hardware efficiency, processor affinity, etc. In a third example, an NN scheduler that may be integrated with a feed-back loop to recognize actual run-time attributes may be used to “learn” and adapt to change its future scheduling behavior. Each of these examples may be integrated individually, or together, to provide an NN scheduler that optimizes and adapts processing functions for an NN model either prior to processing or for just-in-time determination of operation scheduling.
1. A computer-implemented method comprising:
obtaining information describing a first neural network (NN) model in an interoperable data format;
processing the information to determine a plurality of operations representing discrete execution units of the first NN model;
identifying attributes from the information to create descriptor information as annotations describing execution criteria of each operation from the plurality of operations;
obtaining information about hardware functionality and availability for a plurality of hardware processors configured to process the first NN model;
matching a first operation to a first selected hardware processor from the plurality of hardware processors, in part, by comparing the annotations for the first operation with functionality of the plurality of hardware processors;
matching a second operation to a second selected hardware processor from the plurality of hardware processors, in part, by comparing the annotations for the second operation with functionality of the plurality of hardware processors;
creating a first schedule entry and executing the first operation on the first selected hardware processor; and
creating a second schedule entry and executing the second operation on the second selected hardware processor,
wherein hardware performance capabilities of the first selected hardware processor and the second selected hardware processor are different,
at least one of the annotations for the first operation align with capabilities of the first selected hardware processor not available for the second selected hardware processor, and
at least one of the annotations for the second operation align with capabilities of the second selected hardware processor not available for the first selected hardware processor.
2. The computer-implemented method of claim 1 , wherein creating the first and second schedule entries comprises dispatching the first and second schedule entries upon completion of creating schedule entries for the plurality of operations.
3. The computer-implemented method of claim 1 , wherein the first operation is dispatched for execution on the first selected hardware processor prior to creating the second schedule entry for dispatching the second operation.
4. The computer-implemented method of claim 1 , further comprising:
identifying a third operation as having an affinity with the first operation based, in part, on the annotations associated with the first and third operations;
combining the first operation with the third operation to result in a job including both the first operation and the third operation; and
treating the job as the first operation for matching, scheduling, and execution.
5. The computer-implemented method of claim 1 , further comprising:
optimizing the interoperable format in conjunction with processing the information to determine the plurality of operations.
6. The computer-implemented method of claim 1 , further comprising:
determining one or more affinities across a subset of the plurality of operations, the subset representing a group of individual operations;
combining the subset into one or more jobs based on the affinities representing possible efficiencies for execution as determined from the annotations for each individual operation; and
using a composite of the annotations for each individual operation to represent annotations for the one or more jobs for matching, scheduling, and execution, wherein a single job and its corresponding composite annotations represent a single operation with respect to further processing.
7. The computer-implemented method of claim 6 , wherein the one or more affinities represent information indicating each of the group of individual operations requires access to a limited access system resource.
8. The computer-implemented method of claim 7 , wherein the limited access system resource comprises a data resource having limited access across a network of compute resources designated to perform functions for processing the first NN model.
9. The computer-implemented method of claim 1 , further comprising:
converting the interoperable data format into a run-time executable format for a specific compute resource based, in part, on the run-time executable format matching hardware criteria of the specific compute resource.
10. The computer-implemented method of claim 9 , wherein converting comprises cross-compiling to create at least a portion of the run-time executable format.
11. The computer-implemented method of claim 1 , further comprising:
monitoring execution parameters of previously dispatched operations or jobs to collect monitoring information;
updating annotation information based on analysis of the monitoring information to create updated annotation information; and
using the updated annotation information for processing a second NN model.
12. The computer-implemented method of claim 1 , further comprising:
monitoring execution parameters of previously dispatched operations or jobs to collect monitoring information; and
updating scheduling criteria used by a scheduler based on analysis of the monitoring information to adjust future selection of hardware processors.
13. The computer-implemented method of claim 1 , further comprising:
monitoring execution parameters of previously dispatched operations or jobs to collect monitoring information;
updating configuration settings of an adaptive device based on analysis of the monitoring information to create an updated adaptive device; and
using the updated adaptive device for executing at least a portion of a second NN model.
14. A non-transitory computer readable medium comprising computer executable instructions stored thereon that when executed by one or more processing units, perform a method to provide a neural network (NN) operation creation and scheduling function, the method comprising:
obtaining information describing a first neural network (NN) model in an interoperable data format;
processing the information to determine a plurality of operations representing discrete execution units of the first NN model;
identifying attributes from the information to create descriptor information as annotations describing execution criteria of each operation from the plurality of operations;
obtaining information about hardware functionality and availability for a plurality of hardware processors configured to process the first NN model;
matching a first operation to a first selected hardware processor from the plurality of hardware processors, in part, by comparing the annotations for the first operation with functionality of the plurality of hardware processors;
matching a second operation to a second selected hardware processor from the plurality of hardware processors, in part, by comparing the annotations for the second operation with functionality of the plurality of hardware processors;
creating a first schedule entry and executing the first operation on the first selected hardware processor; and
creating a second schedule entry and executing the second operation on the second selected hardware processor,
wherein hardware performance capabilities of the first selected hardware processor and the second selected hardware processor are different,
at least one of the annotations for the first operation align with capabilities of the first selected hardware processor not available for the second selected hardware processor, and
at least one of the annotations for the second operation align with capabilities of the second selected hardware processor not available for the first selected hardware processor.
15. The non-transitory computer readable medium of claim 14 , wherein the method further comprises:
identifying a third operation as having an affinity with the first operation based, in part, on the annotations associated with the first and third operations;
combining the first operation with the third operation to result in a job including both the first operation and the third operation; and
treating the job as the first operation for matching, scheduling, and execution.
16. The non-transitory computer readable medium of claim 14 , wherein the method further comprises:
optimizing the interoperable format in conjunction with processing the information to determine the plurality of operations.
17. The non-transitory computer readable medium of claim 14 , wherein the method further comprises:
determining one or more affinities across a subset of the plurality of operations, the subset representing a group of individual operations;
combining the subset into one or more jobs based on the affinities representing possible efficiencies for execution as determined from the annotations for each individual operation; and
using a composite of the annotations for each individual operation to represent annotations for the one or more jobs for matching, scheduling, and execution, wherein a single job and its corresponding composite annotations represent a single operation with respect to further processing.
18. A computer system, comprising:
a network communications interface;
a memory; and
one or more processing units, communicatively coupled to the memory and the network communications interface, wherein the memory stores instructions, that when executed by the one or more processing units, cause the one or more processing units to provide a neural network (NN) operation creation and scheduling function, the NN operation creation and scheduling function configured to:
obtain information describing a first neural network (NN) model in an interoperable data format;
process the information to determine a plurality of operations representing discrete execution units of the first NN model;
identify attributes from the information to create descriptor information as annotations describing execution criteria of each operation from the plurality of operations;
obtain information about hardware functionality and availability for a plurality of hardware processors configured to process the first NN model;
match a first operation to a first selected hardware processor from the plurality of hardware processors, in part, by comparing the annotations for the first operation with functionality of the plurality of hardware processors;
match a second operation to a second selected hardware processor from the plurality of hardware processors, in part, by comparing the annotations for the second operation with functionality of the plurality of hardware processors;
create a first schedule entry and executing the first operation on the first selected hardware processor; and
create a second schedule entry and executing the second operation on the second selected hardware processor,
wherein hardware performance capabilities of the first selected hardware processor and the second selected hardware processor are different,
at least one of the annotations for the first operation align with capabilities of the first selected hardware processor not available for the second selected hardware processor, and
at least one of the annotations for the second operation align with capabilities of the second selected hardware processor not available for the first selected hardware processor.
19. The computer system of claim 18 , wherein the NN operation creation and scheduling function is further configured to:
identify a third operation as having an affinity with the first operation based, in part, on the annotations associated with the first and third operations;
combine the first operation with the third operation to result in a job including both the first operation and the third operation; and
treat the job as the first operation for matching, scheduling, and execution.
20. The computer system of claim 18 , wherein the NN operation creation and scheduling function is further configured to:
optimize the interoperable format in conjunction with processing the information to determine the plurality of operations.