Compressive sensing based image capture using dynamic masking
Some embodiments provide a novel compressive-sensing image capture device and a method of using data captured by the compressive-sensing image capture device. The novel compressive-sensing image capture device includes an array of sensors for detecting electromagnetic radiation. Each sensor in the sensor array has an associated mask that blocks electromagnetic radiation from portions of the sensor. In some embodiments, an array of passive masks is used to block a particular set of areas of each sensor in the sensor array. In some embodiments, the image capture device also includes an array of lenses corresponding to the sensors of the sensor array such that each sensor receives light that passes through a different lens. Some embodiments of the invention provide a dynamic mask array. In some embodiments, a novel machine trained network is provided that processes image capture data captured by the compressive-sensing image capture device to predict solutions to problems.
1. A method for capturing compressed-image data related to a scene, the method comprising:
directing light captured in a first image capture event towards a set of image-capture sensors;
passing the captured light of the first image capture event through an image-compressing mask in a first configuration before the captured light of the first image capture event reaches the set of image-capture sensors;
modifying the image-compressing mask to produce a second configuration of the image-compressing mask;
directing light captured in a second image capture event towards the set of image-capture sensors; and
passing the captured light of the second image capture event through the image-compressing mask in the second configuration before the captured light of the second image capture event reaches the set of image-capture sensors,
wherein compressed-image values produced by the sensor set are provided to an image processing circuit comprising a machine trained (MT) network of processing nodes for the MT network to identify the content of the compressed-image data related to the scene.
2. The method of claim 1 , wherein the image-compressing mask comprises a fixed masking element associated with the sensors of the set of image-capture sensors.
3. The method of claim 1 , wherein the image-compressing mask comprises a liquid crystal display (LCD) mask comprising a plurality of regions, wherein modifying the image-compressing mask comprises modifying a pattern of occluding and transparent regions of the LCD mask used in the first configuration to produce the second image-compressing mask configuration.
4. The method of claim 3 , wherein the image-compressing mask further comprises a set of lenses, wherein each sensor of the set of image-capture sensors is associated with a particular lens in the set of lenses, each particular lens focusing light passing through the transparent regions of the LCD mask onto the sensor associated with the lens.
5. A method for capturing compressed-image data related to a scene, the method comprising:
directing light captured in a first image capture event towards a set of image-capture sensors;
passing the captured light of the first image capture event through an image-compressing mask in a first configuration before the captured light of the first image capture event reaches the set of image-capture sensors;
modifying the image-compressing mask to produce a second configuration of the image-compressing mask;
directing light captured in a second image capture event towards the set of image-capture sensors; and
passing the captured light of the second image capture event through the image-compressing mask in the second configuration before the captured light of the second image capture event reaches the set of image-capture sensors,
wherein compressed-image values produced by the image-capture sensor set are used to produce an output based on the compressed-image values,
wherein the compressed-image values produced by the set of image-capture sensors are a set of compressed-image coefficients that are output values produced by the set of image-capture sensors and are associated with a set of basis functions, each basis function representing a particular sampling of areas of a scene captured as a compressed image,
wherein each masking pattern produced by the image-compressing mask comprises a plurality of patterns of occluded and transparent portions, each pattern associated with a sensor in the set of image-capture sensors, each pattern physically representing one basis function associated with the sensor,
wherein a set of output values of each sensor in the set of image-capture sensors is a measurement of light incident on the sensor through a set of particular image-compressing mask configurations that is interpreted as a subset of the set of compressed-image coefficients that is equivalent to the result of a projection of an image of the captured scene onto the basis function associated with the sensor for the particular image-compressing mask configuration.
6. The method of claim 5 , wherein the output is an uncompressed image that is based on the compressed-image values.
7. The method of claim 5 , wherein the output is a prediction based on the compressed-image values.
8. The method of claim 5 , wherein the set of image-capture sensors comprises 100 to 10,000 individual sensors and the output values of the set of image-capture sensors are used to reproduce an image equivalent to an image captured by an image capture device with more than 1 million sensors.
9. A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for capturing compressed-image data related to a scene, the program comprising sets of instructions for:
causing light captured in a first image capture event to be directed towards a set of image-capture sensors;
causing the captured light of the first image capture event to be passed through an image-compressing mask in a first configuration before the captured light of the first image capture event reaches the set of image-capture sensors;
modifying the image-compressing mask to produce a second configuration of the image-compressing mask;
causing light captured in a second image capture event to be directed towards the set of image-capture sensors; and
causing the captured light of the second image capture event to be passed through the image-compressing mask in the second configuration before the captured light of the second image capture vent reaches the set of image-capture sensors,
wherein compressed-image values produced by the sensor set are provided to an image processing circuit comprising a machine trained (MT) network of processing nodes for the MT network to identify the content of the compressed-image data related to the scene.
10. The non-transitory machine readable medium of claim 9 , wherein the image-compressing mask comprises a fixed masking element associated with the sensors of the set of image-capture sensors.
11. The non-transitory machine readable medium of claim 9 , wherein the image-compressing mask comprises a liquid crystal display (LCD) mask comprising a plurality of regions, wherein the set of instructions for modifying the image-compressing mask comprises a set of instructions for modifying a pattern of occluding and transparent regions of the LCD mask used in the first configuration to produce the second image-compressing mask configuration.
12. The non-transitory machine readable medium of claim 11 , wherein the image-compressing mask further comprises a set of lenses, wherein each sensor of the set of image-capture sensors is associated with a particular lens in the set of lenses, each particular lens focusing light passing through the transparent regions of the LCD mask onto the sensor associated with the lens.
13. A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for capturing compressed-image data related to a scene, the program comprising sets of instructions for:
causing light captured in a first image capture event to be directed towards a set of image-capture sensors;
causing the captured light of the first image capture event to be passed through an image-compressing mask in a first configuration before the captured light of the first image capture event reaches the set of image-capture sensors;
modifying the image-compressing mask to produce a second configuration of the image-compressing mask;
causing light captured in a second image capture event to be directed towards the set of image-capture sensors; and
causing the captured light of the second image capture event to be passed through the image-compressing mask in the second configuration before the captured light of the second image capture event reaches the set of image-capture sensors,
wherein compressed-image values produced by the image-capture sensor set are used to produce an output based on the compressed-image values,
wherein the compressed-image values produced by the set of image-capture sensors are a set of compressed-image coefficients that are output values produced by the set of image-capture sensors and are associated with a set of basis functions, each basis function representing a particular sampling of areas of a scene captured as a compressed image,
wherein each masking pattern produced by the image-compressing mask comprises a plurality of patterns of occluded and transparent portions, each pattern associated with a sensor in the set of image-capture sensors, each pattern physically representing one basis function associated with the sensor, and
wherein a set of output values of each sensor in the set of image-capture sensors is a measurement of light incident on the sensor through a set of particular image-compressing mask configurations that is interpreted as a subset of the set of compressed-image coefficients that is equivalent to the result of a projection of an image of the captured scene onto the basis function associated with the sensor for the particular image-compressing mask configuration.
14. The non-transitory machine readable medium of claim 13 , wherein the output is an uncompressed image that is based on the compressed-image values.
15. The non-transitory machine readable medium of claim 13 , wherein the output is a prediction based on the compressed-image values.
16. The non-transitory machine readable medium of claim 13 , wherein the set of image-capture sensors comprises 100 to 10,000 individual sensors and the output values of the set of image-capture sensors are used to reproduce an image equivalent to an image captured by an image capture device with more than 1 million sensors.