System and method for runtime optimization of neural networks

1 . A system for optimizing runtime of a neural network (NN), the system comprising at least one processor configured to execute modules of instruction code, whereupon execution of said modules of instruction code, the at least one processor is configured to:

obtain an architecture space of untrained NN architectures, wherein said architectures are capable, upon training, to perform a predefined NN function on a given computing device H;

employ a performance estimator module to evaluate a performance parameter value for NN architectures of the architecture space;

divide the architecture space to a plurality of groups G i based on said evaluated performance parameter value;

for at least one group G i , perform an iterative accuracy prediction process on NN architectures of that group, wherein each iteration comprises: (i) employing a resource allocation module to communicate with computing device H to progressively increase allocation of computation resources of the computing device H (ii) employing a training module to partially train NN architectures of that group based on the allocation, and (iii) using a capacity prediction function f( ), to predict an accuracy metric value;

employ a search module to search among the plurality of groups, to determine a group G* that (a) corresponds to an optimal evaluated performance parameter value, and (b) comprises at least one NN architecture that maintains an accuracy metric value that is at least equal to a reference accuracy metric value;

employ the training module to train at least one NN architecture of the determined group G* to perform the NN function; and

during an inference stage, employ the computing device H to apply the at least one trained architecture on an instance of input data, thereby performing the NN function,

wherein said search comprises: (i) an iterative multiplication search, among the plurality of groups G i , to determine a limit group G L ; and (ii) an iterative binary search, among a subset of groups limited by group G L , to determine group G*.

2 . The system of claim 1 , wherein the at least one processor is further configured to:

predict an accuracy metric value of NN architectures within the determined group G*;

train a predefined K 1 number of NN architectures of the determined group G*, corresponding to top valued accuracy metrics;

measure an accuracy metric value of the K 1 trained NN architectures; and

selecta NN architecture having a highest measured accuracy to perform the predefined NN function.

3 . The system of claim 1 , wherein each iteration of the iterative multiplication search (a) relates to a current group G i , corresponding to a unique range of a performance improvement factor (PIF) PIF i , representing improvement of performance in relation a benchmark performance parameter value, and (b) comprises:

predicting the accuracy metric value of one or more NN architectures within the current group G i ;

if at least one accuracy metric value is at least equal to the reference accuracy value, then moving to a next group G i+1 that corresponds PIF i+1 , where PIF i+1 is a product of multiplication of PIF i by a predetermined multiplication factor; and

determining the first group which does not comprise a NN architecture that maintains an accuracy metric value that is at least equal to the reference accuracy metric value as the limit group G L .

4 . The system of claim 1 , wherein each iteration of the iterative binary search (a) relates to a current group G i , corresponding to a unique range of PIF, PIF i , representing improvement of performance in relation a benchmark performance parameter value, and (b) comprises:

predicting the accuracy metric value of one or more NN architectures within the current group G i ;

if at least one accuracy value is at least equal to the reference accuracy value, then moving to a next group G i+1 that corresponds to a PIF i+1 , where PIF i+1 =2*PIF i ; and

if no accuracy value is at least equal to the reference accuracy value, then moving to a next group G i+1 that corresponds to a PIF i+1 , where PIF i+1 =PIF i /2.

5 . The system of claim 3 , wherein predicting the accuracy metric value comprises employing a capacity prediction function f( ), adapted to compare accuracy of untrained or partially trained NN architectures, such that for each two NN architectures A 1 and A 2 , if an accuracy metric value of A 1 is bigger than an accuracy metric value of A 2 , then f(A 1 ) is bigger than f(A 2 ).

6 . The system of claim 5 , wherein predicting the accuracy metric values of NN architectures of the current group G i is performed iteratively, wherein each iteration of the accuracy metric prediction employs progressively increasing computation resources of the at least one processor.

7 . The system of claim 6 , wherein each iteration of the accuracy metric prediction of NN architectures of the current group G i comprises eliminating a portion of the NN architectures of the current group, according to the predicted accuracy metric values, until a subset K 2 of the NN architectures of the current group G i is obtained.

8 . The system of claim 7 , wherein K 2 is selected from a list consisting of: a predefined percentage of the NN architectures of the current group G i , and a predefined number of NN architectures in the current group G i .

9 . The system of claim 1 , wherein the performance parameter value is selected from a list consisting of latency of the NN architecture, throughput of the NN architecture, power consumption of a computing device for executing the NN architecture, memory consumption of the NN architecture on the computing device, and processing cycles consumed by the computing device for executing the NN architecture.

10 . The system of claim 9 , wherein evaluating a latency performance parameter value of a NN architecture comprises:

feeding the evaluated NN architecture with random inputs;

sampling a number of forward path computations of the untrained architecture on the given computing device;

measuring the latency for each forward path computation; and

averaging the resulting latencies, to produce an evaluated latency value of the NN architecture.

11 . The system of claim 1 , wherein evaluating a performance parameter value of a NN architecture comprises:

providing as a first input, one or more structural data elements, representing a structure of the NN architecture to a machine learning (NIL) based model, trained to evaluate a performance parameter value of the NN architecture based on the first input; and

obtaining from the ML-based model a predicted evaluation of the NN architecture performance parameter value based on the first input.

12 . The system of claim 11 , wherein the at least one processor is further configured to:

provide as a second input, one or more computing device parameters, representing characteristics of a computing device, to the ML-based model; and

obtain from the ML-based model a predicted evaluation of the NN architecture performance parameter value, based on the first input and the second input.

13 . The system of claim 11 , wherein training the ML-based model comprises:

receiving, by the at least one processor, one or more annotated performance data elements, that comprise ground truth information pertaining to at least one performance parameter value of a NN architecture; and

training the ML-based model to predict performance metric values for incoming NN architectures, based on a training dataset comprising the annotated performance data elements.

14 . The system of claim 13 , wherein the ML-based model is an active learning model, and wherein training the ML-based model further comprises:

emitting, by the ML-based model, a request to provide annotated performance data elements that correspond to one or more specific NN architecture; and

receiving, by the at least one processor, one or more annotated performance data elements, according to the request.

15 . A method of optimizing inference of a neural network (NN) on a given computing device H, the method comprising:

obtaining an architecture space representing a plurality of untrained NN architectures;

for one or more NN architectures of the plurality of NN architectures, computing a value of a performance parameter;

dividing the architecture space to a plurality of groups Gi, based on the computed performance parameter value;

for at least one group G i , performing an iterative accuracy prediction process on NN architectures of that group, wherein each iteration comprises (i) communicating with computing device H to progressively increase allocation of computation resources of the computing device H, (ii) partially train NN architectures of that group based on the allocation, and (iii) predicting a value of an inference accuracy metric, using a progressive capacity prediction function fusing the progressively increasing computational resources of computing device H;

selecting an optimal NN architecture A* of the plurality of NN architectures based on (a) the division of the architecture space to groups G i and (b) the at least one predicted value of inference accuracy metric, wherein said selection comprises: performing an iterative search among the plurality of groups Gi, to determine a group G* that (a) corresponds to an optimal performance parameter value, and (b) comprises at least one NN architecture that maintains an accuracy metric value that is at least equal to a reference accuracy metric value, and wherein said iterative search comprises: (i) an iterative multiplication search, among the plurality of groups G i , to determine a limit group G L ; and (ii) an iterative binary search, among a subset of groups limited by group G L , to determine group G*; and

applying, during an inference stage, the trained optimal NN architecture A* to perform the NN function on instances of input data samples.

16 . The method of claim 15 , wherein the NN architectures of the architecture space are: (a) distinguished by at least one structural data element, and (b) are defined as being capable of performing a predetermined NN function after training.

17 . The method of claim 16 , further comprising:

training the optimal NN architecture A* to perform the NN function; and

during an inference stage, applying optimal NN architecture A* to perform the NN function on instances of input data samples.

18 . The method of claim 15 , wherein selecting an optimal NN architecture A* comprises:

receiving a benchmark value of an inference accuracy metric;

performing a search among the plurality of architecture groups Gi, to determine a group G* that (a) corresponds to an optimal value of the performance parameter, and (b) comprises one or more NN architectures that correspond to a predicted inference accuracy metric value that is at least equal to the benchmark value; and

selecting at least one NN architecture from group G* as optimal NN architecture A*.

19 . The method of claim 18 , wherein selecting at least one NN architecture from group G* as optimal NN architecture A* comprises:

training a subset of top-ranking NN architectures of group G*, corresponding to top-ranking predicted inference accuracy metric values;

inferring the subset of NN architectures of group G* on a dataset comprising annotated input samples;

measuring the accuracy metric values according to the annotated input samples; and

selecting optimal NN architecture A* according to the measured accuracy metric values.

20 . The method of claim 15 , wherein selecting an optimal NN architecture A* comprises:

receiving a benchmark value of a performance parameter;

selecting a group G* that corresponds to the performance parameter benchmark value; and

selecting an NN architecture of group G*, corresponding to a top-ranking predicted inference accuracy metric value as the optimal NN architecture A*.

21 . The method of claim 20 , wherein selecting a top-ranking NN architecture of group G* comprises:

training a subset of top-ranking NN architectures of group G*, corresponding to top-ranking predicted inference accuracy metric values;

inferring the subset of NN architectures of group G* on a dataset comprising annotated input samples;

for each NN architecture of the subset, measuring the accuracy metric value according to the annotated input samples; and

selecting the optimal NN architecture A* according to the measured accuracy metric values.