CMOS-assisted inside-out dynamic vision sensor tracking for low power mobile platforms

1. An apparatus, comprising:

a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, wherein the DVS is directed at a first field of view of an operating environment;

a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data, wherein the CMOS image sensor is directed at a second field of view of the operating environment, wherein the second field of view at least partially overlaps with the first field of view and wherein an area of overlap between the first field of view and the second field of view contains objects to be tracked by a visual-inertial simultaneous location and mapping (SLAM) pipeline;

a hybrid feature handler configured to receive, as an input, visual data comprising one or more of a DVS output or a CMOS image sensor output, and provide tracked features obtained from the visual data to the visual-inertial SLAM pipeline for performing inside-out device tracking; and

a sensor scheduler configured to switch off the CMOS image sensor in response to a current value of one or more CMOS control factors,

wherein when the CMOS image sensor is switched off in response to the current value of the one or CMOS control factors, the hybrid feature handler receives visual data from the DVS output.

2. The apparatus of claim 1 , wherein the hybrid feature handler is configured to align and convert DVS outputs and CMOS image sensor outputs to a common representation.

3. The apparatus of claim 1 , wherein the one or more CMOS control factors comprise one or more of a current value of remaining battery power, a power budget for the apparatus, a current value of a representation of robustness of feature tracking, a current value of a tracking accuracy confidence value, an initialization state, or a mode of operation.

4. The apparatus of claim 1 , wherein the one or more CMOS control factors comprise a current value of a tracking accuracy confidence value, the current value of the tracking accuracy confidence value determined based on applying a vector representation of another of the one or more CMOS control factors to a predictive model.

5. The apparatus of claim 4 , wherein the predictive model is trained based on a comparison of inside-out tracking outputs from the visual-inertial SLAM pipeline against a ground truth based on outside-in tracking measurements, and the predictive model predicts a relative error between inside-out tracking outputs from the visual-inertial SLAM pipeline relative to the ground truth.

6. The apparatus of claim 4 , wherein the CMOS image sensor is switched off in response to the current value of the tracking accuracy confidence value falling below a threshold value.

7. The apparatus of claim 6 , wherein the threshold value is adjusted based on one or more of a power consumption history or a power budget for the apparatus.

8. The apparatus of claim 1 , wherein the asynchronous stream of event data comprises an asynchronous stream of binary signals.

9. A method, comprising:

receiving, at a hybrid feature handler of an apparatus, an output of a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, wherein the DVS is directed at a first field of view of an operating environment;

receiving, at the hybrid feature handler, an output of a complementary metal-oxide-semiconductor (CMOS) image sensor, the CMOS image sensor configured to output frames of image data, wherein the CMOS image sensor is directed at a second field of view of the operating environment, wherein the second field of view at least partially overlaps with the first field of view, and wherein an area of overlap between the first field of view and the second field of view contains objects to be tracked by a visual-inertial simultaneous location and mapping (SLAM) pipeline;

determining, by the hybrid feature handler, based on visual data comprising one or more of the output of the DVS or the output of the CMOS image sensor, tracked features;

providing the tracked features obtained from the visual data to the visual-inertial SLAM pipeline for performing inside-out device tracking; and

switching off the CMOS image sensor, by a sensor scheduler, in response to a current value of one or more CMOS control factors,

wherein when the CMOS image sensor is switched off in response to the current value of the one or more CMOS control factors, the hybrid feature handler receives visual data from the DVS output.

10. The method of claim 9 , further comprising:

aligning and converting, by the hybrid feature handler, the output of the DVS and the output of the CMOS image sensor to a common representation.

11. The method of claim 9 , wherein the one or more CMOS control factors comprise one or more of a current value of remaining battery power, a power budget for the apparatus, a current value of a representation of robustness of feature tracking, a current value of a tracking accuracy confidence value, an initialization state, or a mode of operation.

12. The method of claim 9 , wherein the one or more CMOS control factors comprise a current value of a tracking accuracy confidence value, the current value of the tracking accuracy confidence value determined based on applying a vector representation of another of the one or more CMOS control factors to a predictive model.

13. The method of claim 12 , wherein the predictive model is trained based on a comparison of inside-out tracking outputs from the visual-inertial SLAM pipeline against a ground truth based on outside-in tracking measurements, and the predictive model predicts a relative error between inside-out tracking outputs from the visual-inertial SLAM pipeline relative to the ground truth.

14. The method of claim 12 , wherein the CMOS image sensor is switched off in response to the current value of the tracking accuracy confidence value falling below a threshold value.

15. The method of claim 14 , wherein the threshold value is adjusted based on one or more of a power consumption history or a power budget for the apparatus.

16. The method of claim 9 , wherein the asynchronous stream of event data comprises an asynchronous stream of binary signals.

17. A non-transitory computer-readable medium comprising program code, which when executed by a processor, causes an apparatus to:

receive, at a hybrid feature handler of an apparatus, an output of a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, wherein the DVS is directed at a first field of view of an operating environment;

receive, at the hybrid feature handler, an output of a complementary metal-oxide-semiconductor (CMOS) image sensor, the CMOS image sensor configured to output frames of image data, wherein the CMOS image sensor is directed at a second field of view of the operating environment, wherein the second field of view at least partially overlaps with the first field of view, and wherein an area of overlap between the first field of view and the second field of view contains objects to be tracked by a visual-inertial simultaneous location and mapping (SLAM) pipeline;

determine, by the hybrid feature handler, based on visual data comprising one or more of the output of the DVS or the output of the CMOS image sensor, tracked features obtained from the visual data;

provide the tracked features to the SLAM pipeline for performing inside-out device tracking; and

switch off the CMOS image sensor, by a sensor scheduler, in response to a current value of one or more CMOS control factors,

wherein when the CMOS image sensor is switched off in response to the current value of the one or CMOS control factors, the hybrid feature handler receives visual data from the DVS output.

18. The non-transitory computer-readable medium of claim 17 , further comprising program code, which when executed by the processor, causes the apparatus to:

align and convert, by the hybrid feature handler, the output of the DVS and the output of the CMOS image sensor to a common representation.

19. The non-transitory computer-readable medium of claim 17 , wherein the one or more CMOS control factors comprise one or more of a current value of remaining battery power, a power budget for the apparatus, a current value of a representation of robustness of feature tracking, a current value of a tracking accuracy confidence value, an initialization state, or a mode of operation.

20. The non-transitory computer-readable medium of claim 17 , wherein the one or more CMOS control factors comprise a current value of a tracking accuracy confidence value, the current value of the tracking accuracy confidence value determined based on applying a vector representation of another of the one or more CMOS control factors to a predictive model.

21. The non-transitory computer-readable medium of claim 20 , wherein the predictive model is trained based on a comparison of inside-out tracking outputs from the visual-inertial SLAM pipeline against a ground truth based on outside-in tracking measurements, and the predictive model predicts a relative error between inside-out tracking outputs from the visual-inertial SLAM pipeline relative to the ground truth.

22. The non-transitory computer-readable medium of claim 20 , wherein the CMOS image sensor is switched off in response to the current value of the tracking accuracy confidence value falling below a threshold value.

23. The non-transitory computer-readable medium of claim 17 , wherein the asynchronous stream of event data comprises an asynchronous stream of binary signals.