Single instruction set architecture (ISA) format for multiple ISAS in machine learning inference engine
A programmable hardware system for machine learning (ML) includes a core and an inference engine. The core receives commands from a host. The commands are in a first instruction set architecture (ISA) format. The core divides the commands into a first set for performance-critical operations, in the first ISA format, and a second set of performance non-critical operations, in the first ISA format. The core executes the second set to perform the performance non-critical operations of the ML operations and streams the first set to inference engine. The inference engine generates a stream of the first set of commands in a second ISA format based on the first set of commands in the first ISA format. The first set of commands in the second ISA format programs components within the inference engine to execute the ML operations to infer data.
1. A programmable hardware system for machine learning (ML), comprising:
a core configured to
receive a plurality of commands from a host, wherein the plurality of commands is in a first instruction set architecture (ISA) format to coordinate components in the programmable hardware system to perform a ML operation on data to infer via machine learning;
divide the plurality of commands into a first set of commands for performance-critical operations and a second set of commands for performance non-critical operations of the ML operation;
execute the second set of commands to perform the performance non- critical operations of the ML operation;
provide the first set of commands for the performance-critical operations to be executed by an inference engine via a function call;
an instruction streaming engine coupled to the core, wherein the instruction streaming engine is configured to
receive the first set of commands via the function call from the core;
convert and stream the first set of commands in a second ISA format to an inference engine; and
said inference engine configured to
receive the first set of commands in the second ISA format;
execute the first set of commands in the second ISA format to program a plurality of components within the inference engine to perform the ML operation on the data.
2. The programmable hardware system of claim 1 , wherein the plurality of components within the inference engine comprises a plurality of first processing units (PODs) and a plurality of second processing units/elements (PEs), wherein each POD of the plurality of PODs and each PE of the plurality of PEs is coupled to one on-chip memory (OCM), and wherein each POD is configured to perform a dense and/or regular computation task of the ML operation and wherein each PE is configured to perform a sparse and/or irregular computation task of the ML operation.
3. The programmable hardware system of claim 1 , wherein the first set of commands in the second ISA format is configured to program at least a component of the plurality of components to fetch one or more input data streams into its queues and registers associated therewith.
4. The programmable hardware system of claim 1 , wherein the first set of commands in the second ISA format programs a plurality of processing components within the inference engine to perform one or more operations on one or more input data streams.
5. The programmable hardware system of claim 1 , wherein the first set of commands in the second ISA format programs at least one component of the plurality of components to write one or more output data streams associated with its ML operations into its associated on-chip memory (OCM).
6. The programmable hardware system of claim 1 , wherein the first set of commands in the first ISA format has a different format from the second ISA format.
7. The programmable hardware system of claim 1 , wherein the first set of commands in the first ISA format is an asynchronous instruction set, wherein each command of the first set of commands in the first ISA format programs a state machine independent of other state machines and runs asynchronously.
8. The programmable hardware system of claim 1 , wherein instruction set architecture associated with the core is different from instruction set architecture of the inference engine.
9. A programmable hardware system for machine learning (ML), comprising:
a core configured to receive a plurality of commands from a host, wherein the plurality of commands is in a first instruction set architecture (ISA) format, wherein the core is configured to divide the plurality of commands into a first set of commands for performance-critical operations that is in the first ISA format and a second set of commands for performance non-critical operations of ML operations that is in the first ISA format, and wherein the core is further configured to execute the second set of commands to perform the performance non-critical operations of the ML operations and wherein the core is further configured to stream the first set of commands for the performance-critical operations in the first ISA format to an inference engine; and
the inference engine configured to receive the stream of the first set of commands in the first ISA format, wherein the inference engine is further configured to generate a stream of the first set of commands in a second ISA format based on the first set of commands in the first ISA format, wherein the first set of commands in the second ISA format programs a plurality of components within the inference engine to execute the ML operations to infer data.
10. The programmable hardware system of claim 9 further comprising an instruction streaming engine configured to receive the first set of commands in the first ISA format via a function call from the core, and wherein the instruction streaming engine is configured to generate the first set of commands in the second ISA format.
11. The programmable hardware system of claim 9 , wherein the plurality of components within the inference engine comprises a plurality of first processing units (PODs) and a plurality of second processing units/elements (PEs), wherein each POD of the plurality of PODs and each PE of the plurality of PEs is coupled to one on-chip memory (OCM), and wherein each POD is configured to perform a dense and/or regular computation task of an ML operation and wherein each PE is configured to perform a sparse and/or irregular computation task of the ML operation.
12. The programmable hardware system of claim 9 , wherein the first set of commands in the second ISA format is configured to program at least a component of the plurality of components to fetch one or more input data streams into its queues and registers associated therewith.
13. The programmable hardware system of claim 9 , wherein the first set of commands in the second ISA format programs a plurality of processing components within the inference engine to perform one or more operations on one or more input data streams.
14. The programmable hardware system of claim 9 , wherein the first set of commands in the second ISA format programs at least one component of the plurality of components to write one or more output data streams associated with its ML operation into its associated on-chip memory (OCM).
15. The programmable hardware system of claim 9 , wherein the first set of commands in the first ISA format has a different format from the second ISA format.
16. The programmable hardware system of claim 9 , wherein the first set of commands in the first ISA format is an asynchronous instruction set, wherein each command of the first set of commands in the first ISA format programs a state machine independent of other state machines and runs asynchronously.
17. The programmable hardware system of claim 9 , wherein instruction set architecture associated with the core is different from instruction set architecture of the inference engine.
18. A method comprising:
receiving a plurality of commands at a core from a host, wherein the plurality of commands is in a first instruction set architecture (ISA) format to coordinate components in a system to perform a ML operation on data to infer via machine learning;
dividing the plurality of commands at the core, wherein the plurality of commands is divided into a first set of commands for performance-critical operations that is in the first ISA format and a second set of commands for performance non-critical operations of the ML operation;
executing the second set of commands at the core to perform the performance non-critical operations of the ML operations;
streaming the first set of commands for the performance-critical operations in the first ISA format from the core to an instruction streaming engine;
generating and providing by the instruction streaming engine, a stream of the first set of commands in a second ISA format to an inference engine based on the first set of commands in the first ISA format; and
executing by the inference engine the stream of the first set of commands in the second ISA format to program a plurality of components within the inference engine to perform the ML operation on the data.
19. The method of claim 18 , wherein the first set of commands in the first ISA format is received via a function call.
20. The method of claim 18 further comprising programming at least a component of the plurality of components, using the stream of the first set of commands in the second ISA format, to fetch one or more input data streams into its queues and registers associated therewith.
21. The method of claim 18 further comprising programming at least processing component of the plurality of components within the inference engine, using the first set of commands in the second ISA format, to perform one or more operations on one or more input data streams.
22. The method of claim 18 further comprising programming at least one component of the plurality of components, using the first set of commands in the second ISA format, to write one or more output data streams associated with the ML operation into its associated on-chip memory (OCM).
23. The method of claim 18 , wherein the first set of commands in the first ISA format has a different format from the second ISA format.