Camera and method for processing image data
A camera ( 10 ) comprising an image sensor ( 24 ) for capturing image data with a plurality of pixels, a first processing unit ( 30 ) for reading out the image data of the image sensor ( 24 ) in an image data stream comprising groups of adjacent pixels, a second processing unit ( 32 ) for processing the image data, and a memory ( 34 ) for storing at least one of the image data and processing results from the image data, wherein a neural network ( 38 ) is implemented on at least one of the first processing unit ( 30 ) and the second processing unit ( 32 ) and wherein the first processing unit ( 30 ) is configured to already process groups of pixels with at least a first layer ( 42 ) of the neural network while further groups of the image data are still being read out from the image sensor ( 24 ).
1. A camera ( 10 ) comprising:
an image sensor ( 24 ) for capturing image data with a plurality of pixels, a first processing unit ( 30 ) for reading out the image data of the image sensor ( 24 ) in an image data stream comprising groups of adjacent pixels;
a second processing unit ( 32 ) for processing the image data; and
a memory ( 34 ) for storing at least one of the image data and processing results from the image data,
wherein a neural network ( 38 ) comprising a Convolutional Neural Network (CNN) is implemented on at least one of the first processing unit ( 30 ) and the second processing unit ( 32 ),
wherein the first processing unit ( 30 ) is configured to have started processing groups of pixels with at least a first layer ( 42 ) of the neural network while further groups of the image data are still being read out from the image sensor ( 24 ), thereby implementing on-the-fly or pipelined processing of data layers while still processing additional data layers,
and wherein the second processing unit ( 32 ) comprises a classical decoder ( 40 ) operating without the neural network for reading codes ( 20 ), and at least one of further layers ( 44 ) of the neural network and/or a further neural network are implemented on the second processing unit ( 32 ) in order to further process at least one of the image data and the processing results of the first processing unit ( 30 ).
2. The camera ( 10 ) according to claim 1 , wherein the camera ( 10 ) is configured as a camera-based code reader.
3. The camera ( 10 ) according to claim 1 , wherein the group of adjacent pixels is an image line or a part thereof, thus image lines are already processed by the first layer ( 42 , 48 ) while further image lines are still being read out.
4. The camera ( 10 ) according to claim 1 , wherein the first processing unit ( 30 ) comprises a field programmable gate array (FPGA).
5. The camera ( 10 ) according to claim 1 , wherein the second processing unit ( 32 ) comprises a microprocessor.
6. The camera ( 10 ) according to claim 1 , wherein the neural network ( 42 ) is configured for a segmentation of the image data for locating at least one of regions of interest and code regions ( 20 ).
7. The camera ( 10 ) according to claim 1 , wherein the at least first layer ( 42 ) of the neural network on the first processing ( 30 ) generates a feature map which is stored as a processing result in the memory ( 34 ).
8. The camera ( 10 ) according to claim 7 , wherein the feature map comprises a probability map including probability information about locations of codes in the image data.
9. The camera ( 10 ) according to claim 7 , wherein the feature map has a lower resolution than the image data of the image sensor.
10. The camera ( 10 ) according to claim 1 , wherein the first processing unit ( 30 ) comprises at least a first buffer memory ( 50 , 54 ) for temporarily storing processing results of the at least first layer ( 42 ).
11. The camera ( 10 ) according to claim 1 , wherein at least one neural network layer ( 42 ) implemented on the first processing unit ( 30 ) comprises a step size greater than one, i.e. whose convolution kernel is shifted by more than one pixel at a time.
12. The camera ( 10 ) according to claim 1 , wherein at least one neural network layer ( 42 ) implemented on the first processing unit ( 30 ) comprises a dilated convolution kernel that processes pixels that are not directly adjacent.
13. The camera ( 10 ) according to claim 1 , wherein the first processing unit ( 30 ) is configured to store the read-out image data in the memory ( 34 ).
14. The camera ( 10 ) according to claim 1 , wherein the first processing unit ( 30 ) is configured to store the read-out image data in the memory ( 34 ) after preprocessing.
15. The camera ( 10 ) according to claim 1 , wherein the further processing includes reading of codes ( 20 ).
16. The camera ( 10 ) according to claim 1 , wherein at least two neural networks ( 60 , 62 ) are implemented on the second processing unit ( 32 ), each of which further processes processing results of the first processing unit ( 30 ).
17. The camera ( 10 ) according to claim 1 , comprising at least one additional processing unit ( 56 ) for processing steps of the neural network ( 58 ).
18. The camera ( 10 ) according to claim 17 , wherein the additional processing unit ( 56 ) comprises at least one of an NPU (Neural Processing Unit) and a GPU (Graphics Processing Unit).
19. A method for processing image data with a plurality of pixels, comprising the steps of:
reading out the image data from an image sensor ( 24 ) in an image data stream of groups of adjacent pixels using a first processing unit ( 30 );
processing the image data by means of a neural network ( 38 ) comprising a Convolutional Neural Network (CNN) implemented on at least one of the first processing unit ( 30 ) and a second processing unit ( 32 ); and
storing at least one of the image data and processing results in a memory ( 34 ), wherein the first processing unit ( 30 ) starts processing groups of pixels with at least a first layer ( 42 ) of the neural network while further groups of image data are still being read out from the image sensor ( 24 ), thereby implementing on-the-fly or pipelined processing of data layers while still processing additional data layers, and wherein the second processing unit ( 32 ) comprises a classical decoder ( 40 ) operating without the neural network for reading codes ( 20 ), and at least one of further layers ( 44 ) of the neural network and/or a further neural network are implemented on the second processing unit ( 32 ) in order to further process at least one of the image data and the processing results of the first processing unit ( 30 ).
20. The method according to claim 19 , wherein the group of adjacent pixels is an image line or a part thereof, thus image lines are already processed by the first layer ( 42 , 48 ) while further image lines are still being read out.
21. The method according to claim 19 , wherein the neural network ( 42 ) is configured for a segmentation of the image data for locating at least one of regions of interest and code regions ( 20 ).