Low-power, fast-response machine learning autofocus enhancements
Computing devices, such as mobile computing devices, have access to one or more image sensors that can capture images with multiple subjects. Some of these subjects may be known to the user capturing an image with the image sensor. The user may prefer to have the captured image data be optimized around the known subjects. Low-power, fast-response machine learning logic can be configured to allow for the generation of a plurality of inference data. This inference data can be utilized along with other sensor data, such as a motion sensor, for the generation of one or more image sensor configuration changes that may be implemented to optimize the subsequent capture of image data. This cycle of image data analysis, image sensor optimization, and subsequent capture can continue multiple times until a threshold of optimization or time is met. The captured image data optimized around the known subjects is then stored.
1. A device, comprising:
an image sensor;
a Non-Volatile Memory (NVM) for storing data and executable instructions; and
a processor communicatively coupled to the NVM, the processor being configured to direct the device to:
receive image data for processing;
pass the received image data to a machine learning model;
recognize a known subject within the image data;
generate a plurality of inferences via the machine learning model;
utilize the plurality of inferences to generate image optimization data;
wherein the image optimization data comprises one or more image sensor configurations for optimizing an area within the image data associated with the known subject during subsequent image data captures.
2. The device of claim 1 , wherein the processor comprises a machine learning processor comprising a plurality of non-volatile memory cells to store weights for the machine learning model, and wherein the machine learning processor is configured to apply signals corresponding to the received image data, via one or more signal lines associated with the memory cells, to the memory cells, to generate the plurality of inferences.
3. The device of claim 2 , wherein the non-volatile memory cells are Spin-Orbit Torque Magnetoresistive Random-Access Memory (SOT MRAM) memory cells.
4. The device of claim 1 , further comprising a motion sensor, wherein the device is further configured to:
receive motion data from the motion sensor; and
generate the image optimization data based on the plurality of inferences and the received motion data.
5. The device of claim 1 , wherein the device is configured to continue capturing image data until the optimization of the known subject within the image data exceeds a predetermined threshold.
6. The device of claim 5 , wherein the image data is captured in discrete units.
7. The device of claim 6 , wherein the device generates unique image optimization data between each discrete unit of image data captures.
8. The device of claim 5 , wherein the predetermined threshold is related to the focus of the known subject within the image data.
9. The device of claim 1 , wherein the device is further configured to recognize two or more known subjects within the image data and generate image optimization data comprising one or more image sensor configurations for optimizing areas of the image data associated with the two or more known subjects.
10. The device of claim 1 , wherein the area associated with the known subject is a bounding box encasing the known subject.
11. The device of claim 1 , wherein the area associated with the known subject is a mask covering the known subject.
12. A method for generating machine-learning based image optimization data, comprising:
receiving image data;
passing the received image data to a low-power, fast-response machine learning model;
recognizing one or more known subjects within the image data;
determining a known subject area for optimization based on the one or more known subjects;
generating a plurality of inferences via the machine learning model;
utilizing the plurality of inferences to generate image optimization data wherein the image optimization data comprises one or more image sensor configurations for optimizing the known subject area within the image data;
providing the image optimization data to an image sensor for capturing subsequent image data; and
processing subsequently received image data until a predetermined optimization threshold is exceeded.
13. The method of claim 12 , wherein the optimization includes improving the focus of the known subject area.
14. The method of claim 12 , wherein the low-power, fast-response machine learning model is executed in a plurality of Magnetoresistive Random-Access Memory (MRAM) based machine learning devices.
15. The method of claim 14 , wherein the generation of the plurality of inferences is completed in less than one millisecond.
16. A device, comprising:
a Non-Volatile Memory (NVM) for storing data and executable instructions; and
a processor communicatively coupled to the NVM, the processor being configured to direct the device to:
receive image data for processing;
pass the received image data to a machine learning model;
recognize a known subject within the image data;
generate a plurality of inferences via the machine learning model;
utilize the plurality of inferences to generate image optimization data;
wherein the image optimization data comprises one or more image sensor configurations for optimizing image data capture areas associated with the known subject.
17. The device of claim 16 , wherein the processor comprises a machine learning processor comprising a plurality of non-volatile memory cells to store weights for the machine learning model, and wherein the machine learning processor is configured to apply signals corresponding to the received image data, via one or more signal lines associated with the memory cells, to the memory cells, to generate the plurality of inferences.
18. The device of claim 16 , wherein the non-volatile memory cells are Spin-Orbit Torque Magnetoresistive Random-Access Memory (SOT MRAM) memory cells.
19. The device of claim 16 , wherein the image sensor comprises a plurality of varying focal length image sensors.
20. The device of claim 16 , wherein the image sensor further comprises a Light Detection and Ranging (LiDAR) camera.
21. The device of claim 16 , further comprises an image sensor array, wherein the image sensor array is disposed within a mobile computing device.
22. The device of claim 16 , wherein the known subject is selected via an input received by the mobile computing device.