Neuromorphic sensor-based virtual sensor

1 . A method comprising:

receiving, from a neuromorphic sensor, a time series of delta images;

receiving auxiliary data indicating (i) an orientation, location, direction, or speed of the neuromorphic sensor, or (ii) metadata of the neuromorphic sensor, the auxiliary data associated with about a same time as the time series of images;

operating, based on an image of the time series of delta images and auxiliary data of the auxiliary data associated with the image as occurring at about a same time, a first machine learning (ML) model resulting in a low-resolution image; and

operating, based on the low-resolution image, a second ML model resulting in a high-resolution image, the high-resolution image of a type different than that produced by the neuromorphic sensor.

2 . The method of claim 1 , wherein the time series of delta images is a first time series of delta images, the method further comprising:

receiving, from the neuromorphic sensor, a second time series of delta images, a last delta image of the first time series of delta images immediately preceding a first delta image of the second time series of delta images.

3 . The method of claim 2 , further comprising receiving second auxiliary data indicating (i) an orientation, location, direction, or speed of the neuromorphic sensor during capture of the second time series of delta images, or (ii) metadata of the neuromorphic sensor, the second auxiliary data associated with about a same time as the second time series of images.

4 . The method of claim 3 , further comprising:

determining correlation values between pixels of a first image of the first time series of delta images that corresponds to a first earliest time of capture and pixels of a second image of the second time series of delta images that corresponds to a second earliest time of capture; and

wherein the first ML model further operates based on the correlation values.

5 . The method of claim 4 , wherein the first ML model includes first and second encoders in series with each other and first and second decoders that are also connected in series.

6 . The method of claim 5 , wherein the second encoder receives the correlation values.

7 . The method of claim 3 , further comprising:

determining correlation values between pixels of a first image of the first time series of delta images that corresponds to a first earliest time of capture and pixels of a second image of the second time series of delta images that corresponds to a second earliest time of capture; and

wherein the second ML model further operates based on the correlation values.

8 . A system comprising:

a neuromorphic sensor configured to generate a time series of delta images;

processing circuitry configured to receive auxiliary data indicating (i) an orientation, location, direction, or speed of the neuromorphic sensor, or (ii) metadata of the neuromorphic sensor, the auxiliary data associated with about a same time as the time series of images;

operate, based on an image of the time series of delta images and auxiliary data of the auxiliary data associated with the image as occurring at about a same time, a first machine learning (ML) model resulting in a low-resolution image; and

operate, based on the low-resolution image, a second ML model resulting in a high-resolution image, the high-resolution image of a type different than that produced by the neuromorphic sensor.

9 . The system of claim 8 , wherein the time series of delta images is a first time series of delta images, the processing circuitry is further configured to:

receive, from the neuromorphic sensor, a second time series of delta images, a last delta image of the first time series of delta images immediately preceding a first delta image of the second time series of delta images.

10 . The system of claim 9 , wherein the processing circuitry is further configured to receive second auxiliary data indicating (i) an orientation, location, direction, or speed of the neuromorphic sensor during capture of the second time series of delta images, or (ii) metadata of the neuromorphic sensor, the second auxiliary data associated with about a same time as the second time series of images.

11 . The system of claim 10 , wherein the processing circuitry is further configured to:

determine correlation values between pixels of a first image of the first time series of delta images that corresponds to a first earliest time of capture and pixels of a second image of the second time series of delta images that corresponds to a second earliest time of capture; and

wherein the first ML model further operates based on the correlation values.

12 . The system of claim 11 , wherein the first ML model includes first and second encoders in series with each other and first and second decoders that are also connected in series.

13 . The system of claim 12 , wherein the second encoder receives the correlation values.

14 . The system of claim 10 , wherein the processing circuitry is further configured to:

determine correlation values between pixels of a first image of the first time series of delta images that corresponds to a first earliest time of capture and pixels of a second image of the second time series of delta images that corresponds to a second earliest time of capture; and

wherein the second ML model further operates based on the correlation values.

15 . A non-transitory machine readable medium including instructions that, when executed by a machine, cause the machine to perform operations for implementing a virtual image sensor, the operations comprising:

receiving, from a neuromorphic sensor, a time series of delta images;

receiving auxiliary data indicating (i) an orientation, location, direction, or speed of the neuromorphic sensor, or (ii) metadata of the neuromorphic sensor, the auxiliary data associated with about a same time as the time series of images;

operating, based on an image of the time series of delta images and auxiliary data of the auxiliary data associated with the image as occurring at about a same time, a first machine learning (ML) model resulting in a low-resolution image; and

operating, based on the low-resolution image, a second ML model resulting in a high-resolution image, the high-resolution image of a type different than that produced by the neuromorphic sensor.

16 . The non-transitory machine-readable medium of claim 15 , wherein the time series of delta images is a first time series of delta images, the operations further comprising:

receiving, from a neuromorphic sensor, a second time series of delta images, a last delta image of the first time series of delta images immediately preceding a first delta image of the second time series of delta images.

17 . The non-transitory machine-readable medium of claim 16 , wherein the operations further comprise receiving second auxiliary data indicating (i) an orientation, location, direction, or speed of the neuromorphic sensor during capture of the second time series of delta images, or (ii) metadata of the neuromorphic sensor, the second auxiliary data associated with about a same time as the second time series of images.

18 . The non-transitory machine-readable medium of claim 17 , wherein the operations further comprise:

determining correlation values between pixels of a first image of the first time series of delta images that corresponds to a first earliest time of capture and pixels of a second image of the second time series of delta images that corresponds to a second earliest time of capture; and

wherein the first ML model further operates based on the correlation values.

19 . The non-transitory machine-readable medium of claim 18 , wherein the first ML model includes first and second encoders in series with each other and first and second decoders that are also connected in series.

20 . The non-transitory machine-readable medium of claim 19 , wherein the second encoder receives the correlation values.