Machine learning with variable-shape tensors
An apparatus includes a memory and one or more processors. The memory is configured to store tensors for Machine Learning (ML) processing. The one or more processors are configured to receive a work plan associated with a subgraph of a ML graph of a ML model, the work plan supports processing of tensors having respective shapes in a selected range of shapes. A shape of a tensor specifies respective sizes of dimensions of that tensor. The one or more processors are further configured to receive from the memory an input tensor having an actual shape, to modify the work plan based on the actual shape to produce a modified work plan for processing the input tensor in accordance with the subgraph, and to process the input tensor in accordance with the subgraph by submitting the modified work plan for execution by one or more of the processors.
1. An apparatus, comprising:
a memory configured to store tensors for Machine Learning (ML) processing; and
one or more processors, configured to:
receive a work plan associated with a subgraph of a ML graph of a ML model, wherein the work plan supports processing of tensors having respective shapes in a selected range of shapes, and wherein a shape of a tensor specifies respective sizes of dimensions of that tensor;
receive an input tensor that was stored in the memory, the input tensor having an actual shape, wherein the actual shape depends on a non-inferable tensor operation in a previously executed subgraph, wherein the non-inferable tensor operation generates a tensor whose shape depends on actual data in one or more tensors input to the non-inferable tensor operation;
based on the actual shape, modify the work plan to produce a modified work plan for processing the input tensor in accordance with the subgraph; and
process the input tensor in accordance with the subgraph, by submitting the modified work plan for execution by one or more of the processors;
wherein at least one of the one or more processors comprises a ML computational engine that is assigned to execute at least part of the modified work plan.
2. The apparatus according to claim 1 , wherein the one or more processors are configured to receive multiple input tensors for processing using the work plan in a common processing cycle, to modify the work plan based on actual shapes of the respective input tensors for producing the modified work plan, and to processes the multiple input tensors in the common processing cycle using the modified work plan, the input tensors belonging to respective ranges of shapes.
3. The apparatus according to claim 1 , wherein the subgraph comprises one or more tensor operations, and wherein the one or more processors are configured to produce the modified work plan by modifying a tensor operation in the subgraph that receives as input variable-shape tensors, so as to operate on an input tensor having the actual shape.
4. The apparatus according to claim 1 , wherein the one or more processors are configured to determine the selected range of shapes based on shapes of tensors received over multiple iterations of processing the subgraph.
5. The apparatus according to claim 1 , wherein at least one of the one or more processors comprises a Central Processing Unit (CPU), and wherein the one or more processors are configured to assign a portion of the ML graph to the CPU and another portion of the ML graph to the ML computational engine.
6. The apparatus according to claim 1 , wherein the one or more processors are configured to produce, for the subgraph, an updated work plan corresponding to an updated range of shapes different from the selected range of shapes, and to process subsequent input tensors whose shapes belong to the updated range of shapes using the updated work plan.
7. The apparatus according to claim 6 , wherein the updated range of shapes comprises a refined range of shapes in which at least one dimension has a refined range of sizes smaller than a range of sizes of that dimension in the selected range of shapes.
8. The apparatus according to claim 7 , wherein the one or more processors are configured to decide whether to further refine the refined range of shapes based on one or both of (i) shapes of multiple received input tensors that belong to the refined range of shapes, and (ii) an improvement in execution time of the subgraph using the updated work plan exceeding a predefined improvement threshold.
9. The apparatus according to claim 6 , wherein the one or more processors are configured to identify that a given dimension in the actual shape exceeds a corresponding size in the selected range of shapes, to determine the updated range of shapes so as to contain the actual shape, and to process the input tensor using a work plan corresponding to the updated range of shapes.
10. A method, comprising:
receiving by one or more processors a work plan associated with a subgraph of a ML graph of a Machine Learning (ML) model, wherein the work plan supports processing of tensors having respective shapes in a selected range of shapes, and wherein a shape of a tensor specifies respective sizes of dimensions of that tensor;
receiving an input tensor having an actual shape, wherein the actual shape depends on a non-inferable tensor operation in a previously executed subgraph, wherein the non-inferable tensor operation generates a tensor whose shape depends on actual data in one or more tensors input to the non-inferable tensor operation;
based on the actual shape, modifying the work plan to produce a modified work plan for processing the input tensor in accordance with the subgraph; and
processing the input tensor in accordance with the subgraph, by submitting the modified work plan for execution by one or more of the processors;
wherein at least one of the one or more processors comprises a ML computational engine that is assigned to execute at least part of the modified work plan.
11. The method according to claim 10 , and comprising receiving multiple input tensors for processing in a common processing cycle, the input tensors belonging to respective ranges of shapes, modifying the work plan based on actual shapes of the respective input tensors for producing the modified work plan, and processing the multiple input tensors in the common processing cycle using the modified work plan.
12. The method according to claim 10 , wherein the subgraph comprises one or more tensor operations, and wherein producing the modified work plan comprises modifying a tensor operation in the subgraph that receives as input variable-shape tensors, so as to operate on an input tensor having the actual shape.
13. The method according to claim 10 , and comprising determining the selected range of shapes based on shapes of tensors received over multiple iterations of processing the subgraph.
14. The method according to claim 10 , wherein at least one of the one or more processors comprises a Central Processing Unit (CPU), the method further comprising assigning a portion of the ML graph to the CPU and another portion of the ML graph to the ML computational engine.
15. The method according to claim 10 , wherein and comprising producing for the subgraph an updated work plan corresponding to an updated range of shapes different from the selected range of shapes, and processing subsequent input tensors whose shapes belong to the updated range of shapes using the updated work plan.
16. The method according to claim 15 , wherein producing the updated range of shapes comprises producing a refined range of shapes in which at least one dimension has a refined range of sizes smaller than a range of sizes of that dimension in the selected range of shapes.
17. The method according to claim 16 , wherein and comprising deciding whether to further refine the refined range of shapes based on one or both of (i) shapes of multiple received input tensors that belong to the refined range of shapes, and (ii) an improvement in execution time of the subgraph using the updated work plan exceeding a predefined improvement threshold.
18. The method according to claim 15 , and comprising identifying that a given dimension in the actual shape exceeds a corresponding size in the selected range of shapes, determining the updated range of shapes so as to contain the actual shape, and processing the input tensor using a work plan corresponding to the updated range of shapes.
19. An apparatus, comprising:
a memory configured to store tensors for Machine Learning (ML) processing; and
one or more processors, at least one of the one or more processors comprising a ML computational engine, the one or more processors configured to:
receive a work plan associated with a subgraph of a ML graph of a ML model, wherein the work plan supports processing of tensors having respective shapes in a selected range of shapes, and wherein a shape of a tensor specifies respective sizes of dimensions of that tensor;
receive an input tensor that was stored in the memory, the input tensor having an actual shape;
based on the actual shape, modify the work plan to produce a modified work plan for processing the input tensor in accordance with the subgraph; and
process the input tensor in accordance with the subgraph, by submitting the modified work plan for execution by the ML computational engine;
wherein the one or more processors are configured to produce, for the subgraph, an updated work plan corresponding to an updated range of shapes different from the selected range of shapes, and to process subsequent input tensors whose shapes belong to the updated range of shapes using the updated work plan.
20. The apparatus according to claim 19 , wherein the updated range of shapes comprises a refined range of shapes in which at least one dimension has a refined range of sizes smaller than a range of sizes of that dimension in the selected range of shapes.
21. The apparatus according to claim 20 , wherein the one or more processors are configured to decide whether to further refine the refined range of shapes based on one or both of (i) shapes of multiple received input tensors that belong to the refined range of shapes, and (ii) an improvement in execution time of the subgraph using the updated work plan exceeding a predefined improvement threshold.
22. The apparatus according to claim 19 , wherein the one or more processors are configured to identify that a given dimension in the actual shape exceeds a corresponding size in the selected range of shapes, to determine the updated range of shapes so as to contain the actual shape, and to process the input tensor using a work plan corresponding to the updated range of shapes.