Low-power fast-response machine learning variable image compression
Computing devices, such as mobile computing devices, have access to one or more image sensors that can capture images and video with multiple subjects. Some of these subjects may vary in priority for various tasks. It may be desired to increase or decrease the compression on each subject in order to more efficiently store the image data. Low-power, fast-response machine learning logic can be configured to allow for the generation of a plurality of inference data. Inference data can be associated with the type, motion and/or priority of the subjects as desired. This inference data can be utilized along with other subject data to generate one or more variable compression regions within the image data. The image data can be subsequently processed to compress different areas of the image based on a desired application. The variably compressed image can reduce file sizes and allow for more efficient storage and processing.
1. A device, comprising:
an image sensor;
a Non-Volatile Memory (NVM); and
one or more processors communicatively coupled to the NVM, wherein the one or more processors are collectively configured to direct the device to:
receive image data from the image sensor for processing;
pass the received image data to a machine learning model;
recognize a plurality of subjects within the image data;
determine a region for each recognized subject;
classify the recognized subjects into one or more speed classifications;
generate a plurality of compression groups based on the speed classifications;
select a unique level of compression for each of the one or more compression groups;
compress the region of image data associated with each recognized subject according to the selected level of compression for the classified compression group;
compress at least a portion of the remaining image data utilizing a predetermined level of compression, the predetermined level being different from a selected unique level of compression associated with one of the one or more compression groups; and
store the variably compressed image data in the NVM.
2. The device of claim 1 , wherein the one or more compression groups are based on the relative motion of the recognized subjects.
3. The device of claim 1 , wherein the one or more processors comprise a machine learning processor comprising a plurality of non-volatile memory cells configured to store weights for the machine learning model, and wherein the machine learning processor is configured to apply signals corresponding to the received image data, via one or more signal lines associated with the memory cells, to the memory cells, to generate a plurality of inferences for processing the image data.
4. The device of claim 3 , wherein the non-volatile memory cells are Spin-Orbit Torque Magnetoresistive Random-Access Memory (SOT MRAM) memory cells.
5. The device of claim 1 , wherein the classification of the recognized subjects utilizes previously processed image data.
6. The device of claim 1 , wherein the one or more processors are further collectively configured to direct the device to generate subject data for each recognized subject.
7. The device of claim 6 , wherein the subject data comprises subject size, subject motion speed, or subject motion direction.
8. The device of claim 7 , wherein the classification of the recognized subjects utilizes previously generated subject data.
9. The device of claim 1 , wherein the predetermined level of compression is a higher level of compression than the selected unique levels of compression.
10. The device of claim 1 , wherein the determined region is a bounding box encasing the recognized subject.
11. The device of claim 1 , wherein the determined region is a pixel mask covering the recognized subject.
12. The device of claim 1 , wherein the image sensor is disposed on an automobile.
13. The device of claim 12 , wherein the image sensor comprises a plurality of varying focal length image sensors.
14. The device of claim 13 , wherein the image sensor further comprises a Light Detection and Ranging (LiDAR) camera.
15. The device of claim 14 , wherein the recognized subjects are high-priority subjects associated with automobile driving.
16. The device of claim 15 , wherein the high-priority subjects include pedestrians, automobiles, or traffic signs.
17. The device of claim 1 , wherein the variably compressed image data is streamed to a cloud-based computing device.
18. A method for variably compressing image data, comprising:
receiving image data;
passing the received image data to a machine learning model;
processing the image data within the machine learning model to generate a plurality of inferences;
utilizing the generated inferences to:
recognize a plurality of dynamically moving subjects within the image data;
generate a region of image data for each recognized subject;
determine the relative speed of each recognized subject;
generate two or more speed classifications based on the relative speed; and
select a level of compression for each of the plurality of subjects based on the determined speed classifications;
compressing each generated region of image data according to the selected level of compression associated with the corresponding subject; and
compressing at least a portion of the remaining image data utilizing a predetermined level of compression, the predetermined level being different from the selected level of compression for at least one of the plurality of recognized subjects.
19. The method of claim 18 , wherein the machine learning model is processed with a machine learning processor comprising a plurality of non-volatile memory cells to store weights for the machine learning model, and wherein the machine learning processor is configured to apply signals corresponding to the received image data, via one or more signal lines associated with the memory cells, to the memory cells, to generate the plurality of inferences.
20. The method of claim 19 , wherein the non-volatile memory cells are Spin-Orbit Torque Magnetoresistive Random-Access Memory (SOT MRAM) memory cells.
21. The method of claim 18 , wherein the plurality of inferences is generated in less than one millisecond.
22. The method of claim 18 , wherein the selected levels of compression are grouped into one or more categories.
23. The method of claim 22 , wherein the one or more categories are associated with a range of determined speeds and a corresponding compression level.
24. A device, comprising:
one or more processors collectively configured to direct the device to:
receive image data for processing;
pass the received image data to a machine learning model;
recognize two or more subjects within the image data;
classify a first subject based on a first detected speed of the first subject;
classify a second subject based on a second detected speed of the second subject, wherein the second detected speed is different than the first detected speed;
generate a first region of the image data based on the recognized subjects;
generate a second region of the image data comprising the remaining image data;
compress the image data in the first region at a first compression level based on the classifications of the first subject and the second subject; and
compress the image data in the second region at a second compression level different from the first compression level.
25. The device of claim 24 , further comprising a Non-Volatile Memory (NVM) communicatively coupled to the one or more processors, wherein the one or more processors are further collectively configured to cause the device to store the variably compressed image data in the NVM.
26. The device of claim 24 , wherein the one or more processors are further collectively configured to direct the device to send the variably compressed image data to a cloud-based computing device.