Systems and methods for performing direct conversion of image sensor data to image analytics

1. A system for training a generative model that is configured to train a convolution neural network (CNN) to directly process sensor image data, the system comprising:

an image signal processing (ISP) circuitry configure to receive an unlabeled ground truth image in a raw Bayer format and generate an image in a RGB format;

a generative model configured to receive the image in the RGB format and generate an estimated raw image in the raw Bayer format; and

an error generation circuitry configured to receive the unlabeled ground truth image in the raw Bayer format and the estimated raw image and to generate an error between the unlabeled ground truth image and the estimated raw image;

wherein the generative model is configured to train based on a back propagation of the error.

2. The system of claim 1 , wherein the generative model comprises at least one of an autoencoder, a variational autoencoder, or a generative adversarial network (GAN).

3. The system of claim 1 , wherein the error generation circuitry is configured to perform at least one of a loss function, a cross-entropy loss function, or a mean squared loss function.

4. The system of claim 1 , wherein the generative model is configured to train using machine learning.

5. The system of claim 4 , wherein the generative model is configured to train based on the back propagation of the error by updating one or more weights of the generative model.

6. The system of claim 1 , wherein the generative model comprises a convolutional neural network (CNN) having a U-Net architecture.

7. The system of claim 1 :

wherein the generative model comprises a convolutional neural network (CNN) having a modified U-Net architecture comprising an encoder layer and a decoder layer; and

wherein the generative model is configured to generate a scaled input image and pass it to each of the encoder layer and the decoder layer.

8. A method for training a generative model that is configured to train a convolution neural network (CNN) to directly process sensor image data, the method comprising:

receiving an unlabeled ground truth image in a raw Bayer format;

generating an image in a RGB format corresponding to the unlabeled ground truth image;

receiving, at a generative model, the image in the RGB format;

generating, at the generative model, an estimated raw image in the raw Bayer format corresponding to the image in the RGB format;

generating an error between the unlabeled ground truth image and the estimated raw image; and

training, at the generative model, based on a back propagation of the error.

9. The method of claim 8 , wherein the training comprises updating one or more weights of the generative model.

10. A system for training a convolution neural network (CNN) to directly process sensor image data, the system comprising:

an error generation circuitry;

a trained generative model configured to receive an image in an RGB format and generate a corresponding image in a raw Bayer format; and

a CNN configured to receive the image in the raw Bayer format and generate an estimated label;

wherein the error generation circuitry is configured to:

receive a known label corresponding a pattern contained in the image in the RGB format;

receive the estimated label from the CNN; and

generate an error between the estimated label and the known label; and

wherein the CNN is configured to train based on a back propagation of the error.

11. The system of claim 10 , wherein the image in the RGB format is provided with the known label.

12. The system of claim 10 , wherein the image in the raw Bayer format is unlabeled.

13. The system of claim 10 , wherein the error generation circuitry is configured to perform at least one of a loss function, a cross-entropy loss function, or a mean squared loss function.

14. The system of claim 10 , wherein the generative model was trained using machine learning.

15. The system of claim 10 , wherein the CNN is configured to train based on the back propagation of the error by updating one or more weights of the CNN.

16. A method for training a convolution neural network (CNN) to directly process sensor image data, the method comprising:

receiving an image in an RGB format and with a known label;

generating an image in a raw Bayer format corresponding to the image in the RGB format;

generating, at the CNN, an estimated label based on the image in the raw Bayer format;

generating an error between the estimated label and the known label; and

training, at the CNN, based on a back propagation of the error.

17. The method of claim 16 , wherein the training comprises updating one or more weights of the CNN.

18. A method for directly processing sensor image data, the method comprising:

training a generative model to convert an RGB image into an estimated image in a raw Bayer format using a ground truth image in the raw Bayer format;

generating, using the trained generative model, a labeled dataset in the raw Bayer format from a labeled RGB image dataset;

training a convolution neural network (CNN) using the labeled Bayer dataset such that the CNN is configured to directly process sensor images in the raw Bayer format; and

generating, using the trained CNN, image analytics directly from image data in the raw Bayer format captured by a sensor.