System, method and non-transitory computer readable medium for convolutional neural network model optimization targeting blackbox hardware
An object is to provide a tiling based convolutional NN model optimizing system for blackbox hardware. The system including performing means ( 11 ) for changing a size of data and performing a convolutional NN operation included in a first convolutional NN model on the data, managing means ( 12 ) for managing the size of data in association with inference time when the convolutional NN operation is performed on the data, determining ( 13 ) means for determining a timing at which at least one of a tiling operation and a concatenation operation is performed in the first convolutional NN model based on the inference time, and generating means ( 14 ) for generating a second convolutional NN model by adding at least one of the tiling operation and the concatenation operation at the determined timing.
1 . A system comprising:
at least one memory storing instructions; and
at least one processor configured to execute the instructions to:
perform a plurality of test inference runs by repeatedly changing a size of input data for a first convolutional neural network (NN) model and executing a convolutional NN model operation included in the first convolutional NN model on the input data on a blackbox device, wherein internal hardware information of the blackbox device is unknown, to obtain a plurality of measured inference time values corresponding to a plurality of sizes of the input data;
determine, for each operation-axis pair, a threshold size based on a ratio of a change in the measured inference time values to a change in the size of the input data obtained from the plurality of test inference runs;
determine, based on the determined threshold size, one or more layer positions within the first convolutional NN model at which at least one of a tiling operation and a concatenation operation is to be added;
generate a second convolutional NN model by adding the at least one of the tiling operation and the concatenation operation to the first convolutional NN model at the determined one or more layer positions; and
perform an inference operation using the second convolutional NN model on the blackbox device.
2 . The system according to claim 1 , wherein the at least one processor is further configured to execute the instructions to execute the convolutional NN model operation by changing at least one of a height, a width, and a channel of the input data, and manage at least one of the height, the width, and the channel of the input data in association with the measured inference time values.
3 . The system according to claim 1 , wherein the at least one processor is further configured to execute the instructions to determine layer positions at which the tiling operation and the concatenation operation are performed based on an amount of a change in the measured inference time values and an amount of a change in the size of the input data.
4 . The system according to claim 1 , wherein the at least one processor is further configured to execute the instructions to determine layer positions at which the tiling operation and the concatenation operation are performed to reduce the measured inference time values in a case where all convolutional NN operations included in the first convolutional NN model are performed.
5 . A method comprising:
performing a plurality of test inference runs by repeatedly changing a size of input data for a first convolutional neural network (NN) model and executing a convolutional NN model operation included in the first convolutional NN model on the input data on a blackbox device, wherein internal hardware information of the blackbox device is unknown, to obtain a plurality of measured inference time values corresponding to a plurality of sizes of the input data;
determining, for each operation-axis pair, a threshold size based on a ratio of a change in the measured inference time values to a change in the size of the input data obtained from the plurality of test inference runs;
determining, based on the determined threshold size, one or more layer positions within the first convolutional NN model at which at least one of a tiling operation and a concatenation operation is to be added;
generating a second convolutional NN model by adding the at least one of the tiling operation and the concatenation operation to the first convolutional NN model at the determined one or more layer positions; and
performing an inference operation using the second convolutional NN model on the blackbox device.
6 . A non-transitory computer readable medium storing a program for causing a computer to execute:
performing a plurality of test inference runs by repeatedly changing a size of input data for a first convolutional neural network (NN) model and executing a convolutional NN model operation included in the first convolutional NN model on the input data on a blackbox device, wherein internal hardware information of the blackbox device is unknown, to obtain a plurality of measured inference time values corresponding to a plurality of sizes of the input data;
determining, for each operation-axis pair, a threshold size based on a ratio of a change in the measured inference time values to a change in the size of the input data obtained from the plurality of test inference runs;
determining, based on the determined threshold size, one or more layer positions within the first convolutional NN model at which at least one of a tiling operation and a concatenation operation is to be added;
generating a second convolutional NN model by adding the at least one of the tiling operation and the concatenation operation to the first convolutional NN model at the determined one or more layer positions; and
performing an inference operation using the second convolutional NN model on the blackbox device.