Image sensor for processing sensor data to reduce data traffic to host system
Systems, methods and apparatus of integrated image sensing devices. In one example, a system includes a sensor that generates data. A memory device stores the generated data, and further stores a first portion of an artificial neural network (ANN). A host interface of the system is configured to communicate with a host system that stores a second portion of the ANN. The memory device can be stacked with the sensor. The memory device includes an inference engine configured to generate inference results using the stored data as input to the first portion of the ANN. The host interface is further configured to send the inference results to the host system for processing by the host system using the second portion of the ANN.
1. A system comprising:
a first image sensor configured to generate first image data;
at least one second sensor configured to generate sensor data in other than an image form, wherein the sensor data comprises sound data generated by at least one microphone;
at least one controller configured to convert the sound data to second image data using a Fourier transform, wherein the sound data is a time domain signal, and the sound data is converted to a two-dimensional image representation with each position of the two-dimensional image having a value corresponding to a frequency intensity;
a memory device configured to store the generated first image data, and a first portion of an artificial neural network (ANN); and
a host interface configured to communicate with a host system that stores a second portion of the ANN;
wherein the memory device is stacked with the first image sensor, and the memory device comprises an inference engine configured to generate inference results using the stored first image data as input to the first portion of the ANN, and further using the second image data as additional input to the first portion of the ANN;
wherein the host interface is further configured to send the inference results to the host system for processing by the host system using the second portion of the ANN;
wherein the host interface is further configured to receive a write command from the host system to store a third portion of the ANN in the memory device;
wherein the host system is further configured to send the third portion for storing in the memory device in response to evaluating prior data generated by the first image sensor using the second portion of the ANN, and further in response to determining that image frames are being dropped due to bandwidth limitations of communication between the host interface and the host system.
2. The system of claim 1 , further comprising a camera that includes the first image sensor, wherein the first image data is an image stream including frames.
3. The system of claim 2 , wherein:
the first image data is generated by the first image sensor at a rate that exceeds a communication bandwidth between the host interface and the host system; and
the host system accesses the inference results by sending commands to the host interface.
4. The system of claim 1 , wherein:
the write command is a first write command;
the inference engine is a first inference engine and the generated inference results are first inference results;
the controller is configured to send the first inference results to the host system;
the host system comprises a second inference engine configured to generate second inference results from the first inference results; and
the host system sends a second write command executed by the controller to store the second inference results in the memory device.
5. The system of claim 4 , wherein:
the memory device further stores the first inference results; and
the controller is further configured to execute read commands received from the host system via the host interface, the read commands to access the stored first inference results.
6. The system of claim 1 , wherein the host system is further configured to send the third portion to the host interface in response to evaluating inference results received from the host interface.
7. The system of claim 1 , further comprising:
a first sensing device that includes the first image sensor and the memory device; and
a second sensing device that includes the second sensor and the controller.
8. The system of claim 1 , wherein the second sensor includes a sonar sensor.
9. The system of claim 1 , further comprising a package that encapsulates the first image sensor and the memory device.
10. The system of claim 1 , wherein:
the first image sensor is formed on a first integrated circuit die;
the memory device is formed on at least one second integrated circuit die; and
the first integrated circuit die and the at least one second integrated circuit die are stacked to form a three-dimensional integrated circuit.
11. The system of claim 1 , wherein the inference results include at least one of image classification, object detection, or pixel segmentation.
12. The system of claim 11 , wherein the system further comprises:
a control for at least one of steering, braking, or acceleration of a vehicle;
wherein the vehicle generates input for the control based on the inference results from the inference engine.
13. The system of claim 1 , wherein the host interface is further configured to communicate with the host system in accordance with a serial communication protocol.
14. The system of claim 1 , wherein the controller converts the sound data to a spectrogram or heat map.
15. A method comprising:
collecting image data from a first image sensor;
storing, by a memory device, the collected image data;
storing, by the memory device, a first portion of an artificial neural network (ANN);
generating, by a second sensor, sound data;
converting, the sound data to second image data;
communicating, by a host interface, with a host system that stores a second portion of the ANN;
generating, by an inference engine of the memory device, inference results using the stored collected image data as input to the first portion of the ANN, and further using the second image data as additional input to the first portion of the ANN; and
sending, by the host interface, the inference results to the host system for processing by the host system using the second portion of the ANN;
wherein the host interface is further configured to receive a write command from the host system to store a third portion of the ANN in the memory device;
wherein the host system sends the third portion in response to evaluating prior data generated by the first image sensor using the second portion of the ANN, and further in response to determining that image frames are being dropped due to bandwidth limitations of communication between the host interface and the host system.
16. The method of claim 15 , wherein the sound data is converted to the second image data using a Fourier transform.
17. A non-transitory computer-readable medium storing instructions which, when executed on at least one computing device, cause the at least one computing device to:
store, in a memory device, at least a first portion of an artificial neural network (ANN) received from a computing system;
collect image data from an image sensor;
generate inference results using the collected image data as input to the first portion of the ANN;
send, by a communication interface, the inference results to the computing system for processing using a second portion of the ANN stored at the computing system; and
receive a third portion of the ANN from the computing system to store in the memory device, wherein the computing system sends the third portion in response to determining that image frames are being dropped due to bandwidth limitations of communication between the communication interface and the computing system.