High speed computational tracking sensor
A depth measurement assembly (DMA) includes a structured light emitter, an augmented camera, and a controller. The structured light emitter projects structured light into a local area under instructions from the controller. The augmented camera generates image data of an object illuminated with the structured light pattern projected by the structured light emitter in accordance with camera instructions generated by the controller. The augmented camera includes a high speed computation tracking sensor that comprises a plurality of augmented photodetectors. Each augmented photodetector converts light to data and stores the data in its own memory unit. The controller receives the image data and determines depth information of the object in the local area based in part on the image data. The depth measurement unit can be incorporated into a head-mounted display (HMD).
1. A depth measurement unit (DMU) comprising:
a structured light emitter configured to project a structured light pattern into a local area;
an augmented camera comprising a tracking sensor that includes a plurality of augmented photodetectors, each augmented photodetector includes an analog-to-digital converter and a corresponding memory unit, each augmented photodetector configured to convert some of the structured light pattern reflected from an object in the local area to data and store the data in its corresponding memory unit, the tracking sensor configured to operate in a tracking mode, and the augmented camera in the tracking mode configured to:
capture some of the structured light pattern reflected from an object in the local area,
read out data stored in the memory units of a subset of the plurality of augmented photodetectors in accordance with camera instructions, and
generate image data using the data read out from the subset of the plurality of augmented photodetectors; and
a controller configured to:
generate the camera instructions by predicting locations of the structured light pattern in the plurality of augmented photodetectors and selecting the subset of the plurality of augmented photodetectors from the plurality of augmented photodetectors based on the predicted locations, and
determine depth information of the object in the local area based in part on the image data generated using the data read out from the subset of the plurality of augmented photodetectors.
2. The DMU of claim 1 , wherein the memory unit of each augmented photodetector has a same storage size.
3. The DMU of claim 1 , wherein the plurality of augmented photodetectors include at least a first group of augmented photodetectors and a second group of augmented photodetectors, and the memory units of the first group of augmented photodetectors have a smaller storage size than the memory units of the second group of augmented photodetectors.
4. The DMU of claim 3 , wherein the tracking sensor is configured to operate in a full frame mode, and the augmented camera is configured to generate the image data using data read out from all the augmented photodetectors in the first group and all the augmented photodetectors in the second group.
5. The DMU of claim 1 , wherein the tracking sensor is also configured to operate in a full frame mode, and the augmented camera is configured to generate the image data using data read out all of the plurality of augmented photodetectors in the full frame mode.
6. The DMU of claim 5 , wherein the image data from the full frame mode and the tracking mode are interleaved.
7. The DMU of claim 5 , wherein the tracking sensor is configured to operate in the full frame mode periodically.
8. A head-mounted display (HMD) comprising:
a depth measurement unit comprising:
a structured light emitter configured to project a structured light pattern into a local area,
an augmented camera that includes a tracking sensor comprising a plurality of augmented photodetectors, each augmented photodetector includes an analog-to-digital converter and a corresponding memory unit, each augmented photodetector configured to convert some of the structured light pattern reflected from an object in the local area to data and store the data in its corresponding memory unit, the tracking sensor configured to operate in a tracking mode, and the augmented camera in the tracking mode configured to:
capture some of the structured light pattern reflected from an object in the local area,
read out data stored in the memory units of a subset of the plurality of augmented photodetectors in accordance with camera instructions, and
generate image data using the data read out from the subset of the plurality of augmented photodetectors; and
a controller configured to:
generate the camera instructions by predicting locations of the structured light pattern in the plurality of augmented photodetectors and selecting the subset of the plurality of augmented photodetectors from the plurality of augmented photodetectors based on the predicted locations, and
determine depth information of the object in the local area based in part on the image data generated using the data read out from the subset of the plurality of augmented photodetectors;
an electronic display configured to display a virtual object based in part on the depth information; and
an optics block configured to direct light from the electronic display element to an exit pupil of the HMD.
9. The HMD of claim 8 , wherein the memory unit of each augmented photodetector has a same storage size.
10. The HMD of claim 8 , wherein the plurality of augmented photodetectors include at least a first group of augmented photodetectors and a second group of augmented photodetectors, and the memory units of the first group of augmented photodetectors have a smaller storage size than the memory units of the second group of augmented photodetectors.
11. The HMD of claim 8 , wherein the tracking sensor is also configured to operate in a full frame mode, and the augmented camera is configured to generate the image data using data read out from all of the plurality of augmented photodetectors in the full frame mode.
12. An augmented camera comprising:
a tracking sensor comprising a plurality of augmented photodetectors, each augmented photodetector includes an analog-to-digital converter (ADC) and a corresponding memory unit, each augmented photodetector configured to convert light to data and store the data in its corresponding memory unit, and the tracking sensor configured to operate in a tracking mode;
the augmented camera in the tracking mode configured to:
capture light reflected from an object in the local area,
read out data stored in the memory units of a subset of the plurality of augmented photodetectors in accordance with camera instructions, the camera instructions generated by predicting locations of a structured light pattern in the plurality of augmented photodetectors and selecting the subset of the plurality of augmented photodetectors from the plurality of augmented photodetectors based on the predicted locations, and
generate image data using the data read out from the subset of the plurality of augmented photodetectors.
13. The augmented camera of claim 12 , wherein the memory unit of each augmented photodetector has a same storage size.
14. The augmented camera of claim 12 , wherein the plurality of augmented photodetectors include at least a first group of augmented photodetectors and a second group of augmented photodetectors, and the memory units of the first group of augmented photodetectors have a smaller storage size than the memory units of the second group of augmented photodetectors.
15. The augmented camera of claim 12 , wherein the tracking sensor is also configured to operate in a full frame mode, and the augmented camera is configured to generate the image data using data read out from all of the plurality of augmented photodetectors in the full frame mode.
16. The augmented camera of claim 15 , wherein the image data from the full frame mode and the tracking mode are interleaved.
17. The augmented camera of claim 12 , wherein an augmented photodetector is configured to operate under a time-to-saturation scheme and the memory unit of the augmented photodetector is configured to operate as a counter.
18. The augmented camera of claim 12 , wherein each augmented photodetector further includes a corresponding photodiode, the photodiode of an augmented photodetector is on a first layer silicon and the ADC and the memory unit of the augmented photodetector is on a second layer silicon.
19. The augmented camera of claim 12 , wherein each augmented photodetector further includes a light sensitive component configured to convert photons to electrons, the component selected from a group consisting of a thin film and a photodiode.