Neural network transpose layer removal

1 . An apparatus comprising:

a hybrid threading fabric (HTF), wherein the HTF includes single-input multiple-output (SIMD) processor; and

a processor configured to:

access, from memory or storage of the apparatus, a neural network model that includes matrices of synaptic weights arranged in several layers;

detect, in the neural network model, a transposition of a matrix to a fully connected layer, wherein, to detect the transposition of the matrix to the fully connected layer, the processor is configured to:

tabulate, between each of two layers in the neural network model, neuron-to-neuron connections between the two layers, the fully connected layer including neurons where each of the neurons has at least one connection to a neuron of the other layer; and

locate a transpose instruction in the neural network model or a difference in dimensionality between the fully connected layer and the other layer to identify the transposition;

create, in response to detection of the matrix transposition, a modified neural network model, creation of the modified neural network model including:

changes to values of the fully connected layer to correspond to values in the matrix prior to the transposition, wherein, to change the values of the fully connected layer to correspond to values in the matrix prior to the transposition the processor is configured to:

map a first position of the matrix to a second position in the fully connected layer under a flatten operation of the neural network model; and

move a weight, of the fully connected layer, that corresponds to the first position to the second position; and

elimination of the transposition; and

provide the modified neural network model for inference on the HTF, the modified neural network model providing greater inference performance than the neural network model on the HTF due to the elimination of the transposition;

wherein the HTF is configured to run the modified neural network model using the SIMD processor on input data to produce an output with greater performance than running the neural network model on the input to produce the output.

2 . The apparatus of claim 1 , wherein layers of the neural network model prior to the fully connected layer are convolutional neural network layers.

3 . The apparatus of claim 2 , wherein, to move the weight, the processor is configured to compute weights for a new convolution CONV2 of the fully connected layer, which has a first dimension of one and a second dimension of OUTSIZE, from an original convolution CONV1:

for c in range (NCHANNELS):

 for h in range (HEIGHT):

  for w in range (WIDTH):

   for j in range (OUTSIZE):

    i1 = c * HEIGHT * WIDTH + h * WIDTH + w;

    i2 = c * HEIGHT * WIDTH + w * WIDTH + h; and

    CONV2.weight[j][i1] = CONV1.weight[j][i2], wherein

     NCHANNELS, HEIGHT, and WIDTH are values that

     are part of the fully connected layer from the original

     convolution CONV1, and wherein weight[ ][ ] is a

     matrix representation of the fully connected layer.

4 . The apparatus of claim 2 , wherein the convolutional neural network layers were trained separately from layers subsequent to the fully connected layer, and wherein the transposition is a layer incorporated into the neural network model to interface the convolutional neural network layers to the layers subsequent to the fully connected layer.

5 . The apparatus of claim 4 , wherein the convolutional neural network layers are an instance of Multi-Task Cascaded Convolutional Networks (MTCNN) for facial recognition and bounding boxes of faces in images.

6 . The apparatus of claim 1 , wherein, the processor is configured to detect the transposition of the matrix to the fully connected layer and the processor is configured to create the modified neural network model by a compiler targeting the HTF.

7 . The apparatus of claim 1 , wherein the apparatus is a memory device that includes a hybrid threading processor (HTP), a memory controller, and memory.

8 . The apparatus of claim 7 , wherein the processor is the HTP.

9 . The apparatus of claim 7 , wherein, the processor is configured to detect the transposition of the matrix to the fully connected layer and the processor is configured to create the modified neural network model by a just-in-time (JIT) compiler.

10 . The apparatus of claim 7 , wherein the apparatus is configured to:

be included in a vehicle control system that interfaces with a camera of the vehicle configured to capture images as a vehicle operates; and

provide facial recognition and bounding boxes of faces in the images through the inference on the HTF.

11 . The apparatus of claim 1 , wherein the apparatus is a compute-near memory node that includes a host processor and a memory-compute device, wherein the processor is the host processor and wherein the HTF is included in the memory-compute device.

12 . The apparatus of claim 1 , wherein, to provide the modified neural network model for inference on the HTF, the processor is configured to create at least one of an installation package, an executable application, a loadable module, or a script to run on the HTF.

13 . A method comprising:

accessing, from a location memory or storage, a neural network model that includes matrices of synaptic weights arranged in several layers;

detecting, in the neural network model, a transposition of a matrix to a fully connected layer, wherein detecting the transposition of the matrix to the fully connected layer includes:

tabulating, between each of two layers in the neural network model, neuron-to-neuron connections between the two layers, the fully connected layer including neurons where each of the neurons has at least one connection to a neuron of the other layer; and

locating a transpose instruction in the neural network model or a difference in dimensionality between the fully connected layer and the other layer to identify the transposition;

creating, in response to detecting the matrix transposition, a modified neural network model, including:

changing values of the fully connected layer to correspond to values in the matrix prior to the transposition, wherein changing the values of the fully connected layer to correspond to values in the matrix prior to the transposition includes:

mapping a first position of the matrix to a second position in the fully connected layer under a flatten operation of the neural network model; and

moving a weight, of the fully connected layer, that corresponds to the first position to the second position; and

eliminating the transposition;

providing the modified neural network model for inference on computer hardware, the modified neural network model providing greater inference performance than the neural network model on the computer hardware due to elimination of the transposition, the computer hardware including a single-input multiple-output (SIMD) processor; and

running, on the SIMD processor, the modified neural network model on input data to produce an output with greater performance than running the neural network model on the input to produce the output.

14 . The method of claim 13 , wherein layers of the neural network model prior to the fully connected layer are convolutional neural network layers.

15 . The method of claim 14 , wherein moving the weight includes computing weights for a new convolution CONV2 of the fully connected layer, which has a first dimension of one and a second dimension of OUTSIZE, from an original convolution CONV1:

for c in range (NCHANNELS):

 for h in range (HEIGHT):

  for w in range (WIDTH):

   for j in range (OUTSIZE):

    i1 = c * HEIGHT * WIDTH + h * WIDTH + w;

    i2 = c * HEIGHT * WIDTH + w * WIDTH + h; and

    CONV2.weight[j][i1] = CONV1.weight[j][i2], wherein

     NCHANNELS, HEIGHT, and WIDTH are values that

     are part of the fully connected layer from the original

     convolution CONV1, and wherein weight[ ][ ] is a

     matrix representation of the fully connected layer.

16 . The method of claim 13 , wherein the SIMD processor is included in a memory device that also includes a hybrid threading processor (HTP), a memory controller, and memory.

17 . The method of claim 16 , wherein the memory device is configured to:

be included in a vehicle control system that interfaces with a camera of the vehicle configured to capture images as a vehicle operates; and

provide facial recognition and bounding boxes of faces in the images through the inference on the computer hardware.

18 . The method of claim 13 , wherein providing the modified neural network model for inference on the computer hardware includes creating at least one of an installation package, an executable application, a loadable module, or a script to run on the computer hardware.

19 . Non-transitory machine readable media including instructions that, when executed by processing circuitry, cause processing circuitry to perform operations comprising:

accessing, from a location memory or storage, a neural network model that includes matrices of synaptic weights arranged in several layers;

detecting, in the neural network model, a transposition of a matrix to a fully connected layer, wherein detecting the transposition of the matrix to the fully connected layer includes:

tabulating, between each of two layers in the neural network model, neuron-to-neuron connections between the two layers, the fully connected layer including neurons where each of the neurons has at least one connection to a neuron of the other layer; and

locating a transpose instruction in the neural network model or a difference in dimensionality between the fully connected layer and the other layer to identify the transposition;

creating, in response to detecting the matrix transposition, a modified neural network model, including:

changing values of the fully connected layer to correspond to values in the matrix prior to the transposition, wherein changing the values of the fully connected layer to correspond to values in the matrix prior to the transposition includes:

mapping a first position of the matrix to a second position in the fully connected layer under a flatten operation of the neural network model; and

moving a weight, of the fully connected layer, that corresponds to the first position to the second position; and

eliminating the transposition;

providing the modified neural network model for inference on computer hardware, the modified neural network model providing greater inference performance than the neural network model on the computer hardware due to the elimination of the transposition, the computer hardware including a single-input multiple-output (SIMD) processor; and

running, on the SIMD processor, the modified neural network model on input data to produce an output with greater performance than running the neural network model on the input to produce the output.

20 . The non-transitory machine readable media of claim 19 , wherein layers of the neural network model prior to the fully connected layer are convolutional neural network layers.

21 . The non-transitory machine readable media of claim 20 , wherein moving the weight includes computing weights for a new convolution CONV2 of the fully connected layer, which has a first dimension of one and a second dimension of OUTSIZE, from an original convolution CONV1:

for c in range (NCHANNELS):

 for h in range (HEIGHT):

  for w in range (WIDTH):

   for j in range (OUTSIZE):

    i1 = c * HEIGHT * WIDTH + h * WIDTH + w;

    i2 = c * HEIGHT * WIDTH + w * WIDTH + h; and

    CONV2.weight[j][i1] = CONV1.weight[j][i2], wherein

     NCHANNELS, HEIGHT, and WIDTH are values that

     are part of the fully connected layer from the original

     convolution CONV1, and wherein weight[ ][ ] is a

     matrix representation of the fully connected layer.

22 . The non-transitory machine readable media of claim 20 , wherein the convolutional neural network layers were trained separately from layers subsequent to the fully connected layer, and wherein the transposition is a layer incorporated into the neural network model to interface the convolutional neural network layers to the layers subsequent to the fully connected layer.

23 . The non-transitory machine readable media of claim 22 , wherein the convolutional neural network layers are an instance of Multi-Task Cascaded Convolutional Networks (MTCNN) for facial recognition and bounding boxes of faces in images.

24 . The non-transitory machine readable media of claim 19 , wherein detecting the transposition of the matrix to the fully connected layer and creating the modified neural network model are performed by a compiler targeting the computer hardware.

25 . The non-transitory machine readable media of claim 19 , wherein the computer hardware is a hybrid threading fabric (HTF) device.

26 . The non-transitory machine readable media of claim 25 , wherein the HTF device is included in a memory device that also includes a hybrid threading processor (HTP), a memory controller, and memory.

27 . The non-transitory machine readable media of claim 26 , wherein detecting the transposition of the matrix to the fully connected layer and creating the modified neural network model are performed by a just-in-time (JIT) compiler of the memory device.

28 . The non-transitory machine readable media of claim 26 , wherein the memory device is configured to:

be included in a vehicle control system that interfaces with a camera of the vehicle configured to capture images as a vehicle operates; and

provide facial recognition and bounding boxes of faces in the images through the inference on the computer hardware.

29 . The non-transitory machine readable media of claim 19 , wherein providing the modified neural network model for inference on the computer hardware includes creating at least one of an installation package, an executable application, a loadable module, or a script to run on the computer hardware.