Sensor system based on stacked sensor layers
A sensor assembly for determining one or more features of a local area is presented herein. The sensor assembly includes a plurality of stacked sensor layers. A first sensor layer of the plurality of stacked sensor layers located on top of the sensor assembly includes an array of pixels. The top sensor layer can be configured to capture one or more images of light reflected from one or more objects in the local area. The sensor assembly further includes one or more sensor layers located beneath the top sensor layer. The one or more sensor layers can be configured to process data related to the captured one or more images. A plurality of sensor assemblies can be integrated into an artificial reality system, e.g., a head-mounted display.
1. An apparatus comprising:
an array of pixel cells configured to capture one or more first images; and
a processing circuit configured to:
extract features of one or more objects from the one or more first images; and
enable a subset of the array of pixel cells based on the features to capture one or more second images of the one or more objects.
2. The apparatus of claim 1 , wherein the processing circuit is configured to:
transmit first information related to the features to a host device;
receive, from the host device, second information related to the subset of the array of pixel cells; and
enable the subset of the array of pixel cells based on the second information.
3. The apparatus of claim 2 , wherein the first information includes one or more pixel locations of the features in the one or more first images; and
wherein the second information includes expected pixel locations of the features in the one or more second images.
4. The apparatus of claim 2 , wherein the processing circuit is configured to receive, from the host device, third information indicating expected pixel locations of the features in the one or more first images; and
extract the features from the one or more first images based on the third information.
5. The apparatus of claim 2 , wherein the processing circuit is configured to receive the first information at a first frame rate and to transmit the second information at a second frame rate; and
wherein the first frame rate is lower than the second frame rate.
6. The apparatus of claim 1 , wherein the processing circuit is configured to extract the features from the one or more first images based on at least one of: extracting dots based on a centroid estimation operation, or extracting keys or events based on a threshold detection operation.
7. The apparatus of claim 1 , wherein the processing circuit is configured to:
perform a convolution operation on the one or more first images to generate filtered one or more first images; and
extract the features from the filtered one or more first images.
8. The apparatus of claim 1 , wherein the processing circuit is configured to determine depth information for the one or more objects based on the features.
9. The apparatus of claim 1 , wherein the processing circuit is configured to determine content information based on the features.
10. The apparatus of claim 1 , wherein the processing circuit is further configured to implement a convolutional neural network (CNN).
11. The apparatus of claim 10 , wherein the CNN is configured to determine whether the one or more first images include the one or more objects based on the features.
12. The apparatus of claim 10 , wherein the CNN is configured to perform a convolution operation on the one or more first images.
13. The apparatus of claim 10 , wherein the processing circuit comprises an array of memristors configured to store a set of neural network weights of the CNN.
14. The apparatus of claim 1 , wherein the array of pixel cells is formed in a sensor layer;
wherein the processing circuit is formed in one or more processing circuit layers; and
wherein the sensor layer and the one or more processing circuit layers form a stack structure.
15. The apparatus of claim 14 , wherein the sensor layer comprises at least one of: a Quantum Dot (QD) photodetector material, or an organic photonic film (OPF) photodetector material.
16. The apparatus of claim 14 , wherein the one or more processing circuit layers comprises a first processing circuit layer and a second processing circuit layer forming part of the stack structure, the first processing circuit layer including a feature extraction circuit, the second processing circuit layer including a CNN;
wherein an interface connection between the sensor layer and the first processing circuit layer comprises copper pad connections; and
wherein the first processing circuit layer and the second processing circuit layer are electrically coupled using through silicon vias (TSV).
17. A mobile device comprising:
an sensor assembly comprising:
an array of pixel cells configured to capture one or more first images; and
a processing circuit configured to:
extract features of one or more objects from the one or more first images; and
enable a subset of the array of pixel cells based on the features to capture one or more second images of the one or more objects; and
an output device configured to output content, the content being generated based on the features.
18. The mobile device of claim 17 , wherein the output device comprises an optical assembly configured to direct image light of the content to an eye box corresponding to a location of a user's eye.
19. The mobile device of claim 17 , wherein the processing circuit further comprises a first circuit configured to implement a convolutional neural network (CNN).
20. The mobile device of claim 17 , wherein the processing circuit is configured to:
transmit first information related to the features to a host device;
receive, from the host device, second information related to the subset of the array of pixel cells; and
enable the subset of the array of pixel cells based on the second information.