Convolutional neural network with multiple output frames

1. An apparatus comprising:

an interface configured to receive pixel data; and

a processor configured to (i) process said pixel data arranged as a plurality of input image frames and (ii) store and execute a convolutional neural network that simultaneously generates a plurality of output data in response to said plurality of input image frames, wherein

(a) said convolutional neural network comprises (i) an input to simultaneously receive said plurality of input image frames, (ii) an output configured to simultaneously present said plurality of output data, and (iii) a plurality of convolutional layers,

(b) a first layer of said plurality of convolutional layers is configured to simultaneously process said plurality of input image frames,

(c) a last layer of said plurality of convolutional layers is configured to simultaneously generate said plurality of output data,

(d) each of said plurality of input image frames corresponds to a first subset of instances of time,

(e) each of said plurality of output data represents a second subset of instances of time, and

(f) a number of computations by said processor is reduced, compared to generating each of said plurality of output data for said second subset of instances of time sequentially, by generating each of said plurality of output data for said second subset of instances of time simultaneously from said plurality of input image frames from said first subset of instances of time when an increase in said computations for generating said output data for said second subset of instances of time simultaneously is less than a multiple of a number of said computations to generate said output data for one of said instances of time from said second subset.

2. The apparatus according to claim 1 , wherein each of said plurality of output data comprises a predetermined number of planes.

3. The apparatus according to claim 1 , wherein:

said plurality of input image frames comprises a first number of input image frames;

said plurality of output data comprises a second number of output data; and

said first number is equal to said second number.

4. The apparatus according to claim 1 , wherein:

said plurality of input image frames comprises a first number of input image frames;

said plurality of output data comprises a second number of output data; and

said first number is greater than said second number.

5. The apparatus according to claim 1 , wherein each of plurality of said input image frames comprises a Bayer pattern frame and each of said plurality of output data comprises at least one of RGB frames or YUV frames.

6. The apparatus according to claim 1 , wherein said plurality of input image frames comprises a first time series of images and said plurality of output data comprises a second time series of detections.

7. The apparatus according to claim 6 , wherein said detections comprise objects detected and locations.

8. The apparatus according to claim 7 , wherein each of said plurality of output data comprises a separate plane for each object type.

9. The apparatus according to claim 1 , wherein said plurality of input image frames comprises a first time series of images and said plurality of output data comprises a second time series of segmentation maps.

10. The apparatus according to claim 1 , wherein said convolutional neural network is implemented using a UNet architecture comprising a number of expansion blocks equal to a number of contraction blocks.

11. A method of processing images using a convolutional neural network comprising:

receiving pixel data;

processing said pixel data arranged as a plurality of input image frames; and

simultaneously generating a plurality of output data in response to said plurality of input image frames, using said convolutional neural network, wherein

(a) said convolutional neural network comprises (i) an input to simultaneously receive said plurality of input image frames, (ii) an output configured to simultaneously present said plurality of output data, and (iii) a plurality of convolutional layers,

(b) a first layer of said plurality of convolutional layers is configured to simultaneously process said plurality of input image frames,

(c) a last convolution layer of said plurality of convolutional layers is configured to simultaneously generate said plurality of output data,

(d) each of said plurality of input image frames corresponds to a first subset of instances of time,

(e) each of said plurality of output data represents a second subset of instances of time, and

(f) a number of computations by for said convolutional neural network is reduced, compared to generating each of said plurality of output data for said second subset of instances of time sequentially, by generating each of said plurality of output data for said second subset of instances of time simultaneously from said plurality of input image frames from said first subset of instances of time when an increase in said computations for generating said output data for said second subset of instances of time simultaneously is less than a multiple of a number of said computations to generate said output data for one of said instances of time from said second subset.

12. The method according to claim 11 , wherein each of said plurality of output data comprises a predetermined number of planes.

13. The method according to claim 11 , wherein:

said plurality of input image frames comprises a first number of input image frames;

said plurality of output data comprises a second number of output data; and

said first number is equal to said second number.

14. The method according to claim 11 , wherein:

said plurality of input image frames comprises a first number of input image frames;

said plurality of output data comprises a second number of output data; and

said first number is greater than said second number.

15. The method according to claim 11 , further comprising:

receiving said pixel data comprising a plurality of Bayer pattern frames;

converting each of the plurality of Bayer pattern frames to a first number of input planes having a reduced spatial resolution;

performing a first convolution on said first number of input planes using said first layer of said plurality of convolutional layers;

processing a result of said first layer of said plurality of convolutional layers using a fully convolutional network;

generating a second number of output planes having said reduced spatial resolution by performing a last convolution on an output of said fully convolutional network using said last layer of said plurality of convolutional layers; and

generating said plurality of output data from said second number of output planes having said reduced spatial resolution by performing a sub-pixel convolution to obtain at least one of a plurality of RGB frames or a plurality of YUV frames.

16. The method according to claim 15 , wherein said fully convolutional network is implemented using a UNet architecture comprising a number of expansion blocks equal to a number of contraction blocks.

17. The method according to claim 11 , wherein said plurality of input image frames comprises a first time series of images, said plurality of output data comprises a second time series of detections, and said detections comprise objects detected and locations.

18. The method according to claim 17 , wherein each of said plurality of output data comprises a separate plane for each object type.

19. The apparatus according to claim 1 , wherein said convolutional neural network is part of a video processing pipeline.

20. The apparatus according to claim 1 , wherein said convolutional neural network is executed using a plurality of dedicated hardware engines configured by a scheduler to reduce computational complexity and power consumption.