On-device inference method for multi-frame processing in a neural network
A method for optimizing multi-frame processing model of a neural network includes: receiving a plurality of input frames by a processing engine that is configured to execute a multi frame processing model, the multi frame processing model including a plurality of convolution layers; selecting a pre-determined number of frames from the received plurality of frames for processing by the plurality of convolution layers; determining, as a sequence of frames, at least a preceding frame and a plurality of following frames amongst the selected pre-determined number of frames; removing the preceding frame by processing the sequence of frames using a plurality of filters in the multi frame processing model; and concatenating the plurality of following frames in an order, to the plurality of input frames for subsequent receiving by the multi frame processing model.
1 . A method for optimizing multi-frame processing model of a neural network, the method comprising: receiving a first set of concatenated frames and a plurality of previously processed input frames by a processing engine that is configured to execute a multi frame processing model, the first set of concatenated frames including a current input frame; generating a second set of concatenated frames by discarding an oldest previously processed input frame of the plurality of previously processed input frames that is concatenated within the first set of concatenated frames; and providing the second set of concatenated frames to the multi frame processing model for concatenating the second set of concatenated frames with the current input frame, so that the multi frame processing model is optimized by eliminating processing of the discarded oldest previously processed input frame.
2 . The method as claimed in claim 1 ,
wherein the multi frame processing model includes a plurality of predetermined filters, and
wherein the generating the second set of concatenated frames comprises:
performing one or more convolution operations on the first set of concatenated frames using one or more filters of the plurality of predetermined filters; and
discarding the oldest previously processed input frame based on a result of the performing the one or more convolution operations.
3 . The method as claimed in claim 2 , wherein the one or more convolution operations is performed using all of the plurality of predetermined filters.
4 . The method as claimed in claim 2 ,
wherein a filter weight of each of the plurality of predetermined filters corresponds to a dummy weight, and
wherein the dummy weight is associated with each of the plurality of predetermined filters via a dummy weight interleaving method.
5 . The method as claimed in claim 2 , further comprising:
downsampling the current input frame and each of the second set of concatenated frames to a specific resolution; and
generating a third set of concatenated frames by concatenating the down sampled current input frame and the down sampled second set of concatenated frames.
6 . The method as claimed in claim 5 , wherein the downsampling the current input frame to the specific resolution comprises performing a convolution on the current input frame using a filter weight of the one or more filters.
7 . The method as claimed in claim 5 ,
wherein the multi frame processing model further includes a plurality of convolution layers, and
wherein the method further comprises adjusting a weight of a convolution layer of the plurality of convolution layers that receives the first set of concatenated frames based on a change in the specific resolution and a change in an input channel.
8 . The method as claimed in claim 1 ,
wherein the multi frame processing model includes a clipping layer, and
wherein the method further comprises:
determining a clipping range of the clipping layer based on a bit depth and an input range associated with one or more input frames within the second set of concatenated frames; and
limiting one or more outputs of the processing engine to a pre-determined range based on the determined clipping range of the clipping layer.
9 . The method as claimed in claim 1 , wherein the processing engine is a component of the neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural networks (DNN).
10 . An apparatus for optimizing multi-frame processing model of a neural network, the apparatus comprising: a processing engine that is configured to execute a multi frame processing model, the processing engine being further configured to: receive a first set of concatenated frames and a plurality of previously processed input frames, the first set of concatenated frames including a current input frame; generate a second set of concatenated frames by discarding an oldest previously processed input frame of the plurality of previously processed input frames that is concatenated within the first set of concatenated frames; and provide the second set of concatenated frames to the multi frame processing model for concatenating the second set of concatenated frames with the current input frame, so that the multi frame processing model is optimized by eliminating processing of the discarded oldest previously processed input frame.
11 . The apparatus as claimed in claim 10 ,
wherein the multi frame processing model includes a plurality of predetermined filters, and
wherein the processing engine being configured to generate the second set of concatenated frames includes being configured to:
perform one or more convolution operations on the first set of concatenated frames using one or more filters of the plurality of predetermined filters; and
discard the oldest previously processed input frame based on a result of the performed one or more convolution operations.
12 . The apparatus as claimed in claim 11 , wherein the one or more convolution operations is performed using all of the plurality of predetermined filters.
13 . The apparatus as claimed in claim 11 ,
wherein a filter weight of each of the plurality of predetermined filters corresponds to a dummy weight, and
wherein the dummy weight is associated with each of the plurality of predetermined filters via a dummy weight interleaving method.
14 . The apparatus as claimed in claim 11 , wherein the processing engine is further configured to:
downsample the current input frame and each of the second set of concatenated frames to a specific resolution; and
generate a third set of concatenated frames by concatenating the down sampled current input frame and the down sampled second set of concatenated frames.
15 . The apparatus as claimed in claim 14 , wherein the processing engine being configured to downsample the current input frame to the specific resolution includes being configured to perform a convolution on the current input frame using a filter weight of the one or more filters.
16 . The apparatus as claimed in claim 14 ,
wherein the multi frame processing model further includes a plurality of convolution layers, and
wherein the processing engine is further configured to adjust a weight of a convolution layer of the plurality of convolution layers that receives the first set of concatenated frames based on a change in the specific resolution and a change in an input channel.
17 . The apparatus as claimed in claim 11 ,
wherein the multi frame processing model includes a clipping layer, and
wherein the processing engine is further configured to:
determine a clipping range of the clipping layer based on a bit depth and an input range associated with one or more input frames within the second set of concatenated frames; and
limit one or more outputs of the processing engine to a pre-determined range based on the determined clipping range of the clipping layer.
18 . The apparatus as claimed in claim 10 , wherein the processing engine is a component of the neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural networks (DNN).