Semantic mapping for low-power augmented reality using dynamic vision sensor

1. An apparatus comprising:

a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data;

a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data;

an inertial measurement unit (IMU);

a processor; and

a memory containing instructions, which when executed by the processor, cause the apparatus to:

generate a semantic segmentation of a time-stamped frame, the time-stamped frame selectively based on: 1) one of an output of the CMOS image sensor or a synthesized event frame, or 2) both of the output of the CMOS image sensor and the synthesized event frame, wherein the synthesized event frame is based on an output from the DVS and an output from the IMU over a time interval, and wherein the semantic segmentation comprises a semantic label associated with a region of the time-stamped frame,

determine, based on the semantic segmentation, a simplified object representation in a coordinate space, and

update a stable semantic map based on the simplified object representation.

2. The apparatus of claim 1 , wherein the memory further contains instructions, which, when executed by the processor, further cause the apparatus to:

generate the semantic segmentation of the time-stamped frame by passing the output of the CMOS image sensor and the synthesized event frame through a deep convolutional neural network (CNN) to apply a semantic label to at least one pixel of the time-stamped frame.

3. The apparatus of claim 1 , further comprising a sensor scheduler configured to switch off the CMOS image sensor based on a current value of one or more CMOS control factors.

4. The apparatus of claim 1 , wherein the memory further contains instructions, which, when executed by the processor, further cause the apparatus to:

receive, from a visual-inertial simultaneous location and mapping (SLAM) pipeline, pose data of the apparatus, the pose data associated with the time-stamped frame,

receive, from the visual-inertial SLAM pipeline, point cloud data associated with the time-stamped frame, and

determine the simplified object representation, based on the pose data and the point cloud data, wherein the simplified object representation comprises a higher-level geometric primitive associated with the semantic label.

5. The apparatus of claim 4 , wherein the higher-level geometric primitive is a plane, and the memory further contains instructions, which, when executed by the processor, further cause the apparatus to:

receive gravity data associated with a three-dimensional environment represented in part by the point cloud data,

cluster the point cloud data into a bounded planar region,

parameterize the bounded planar region,

label the bounded planar region based on the semantic label, and

output a labeled plane.

6. The apparatus of claim 4 , wherein the higher-level geometric primitive is a bounding box, and the memory further contains instructions, which, when executed by the processor, further cause the apparatus to:

receive gravity data associated with a three-dimensional environment represented in part by the point cloud data,

cluster the point cloud data associated with the semantic label,

determine a label for the clustered point cloud data,

determine coordinates of a minimum three-dimensional bounding box enclosing the clustered point cloud data, and

output a labeled bounding box.

7. The apparatus of claim 1 , wherein the memory further contains instructions, which, when executed by the processor, further cause the apparatus to:

associate the simplified object representation with an element set of the stable semantic map,

determine a value of a confidence interval of the element set, and

responsive to the value of the confidence interval of the element set exceeding a threshold, update the stable semantic map by merging the simplified object representation into the element set.

8. A computer-implemented method, comprising:

generating, at an apparatus comprising a processor and a memory, a semantic segmentation of a time-stamped frame,

wherein the time-stamped frame is selectively based on: 1) one of an output of a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data or a synthesized event frame, or 2) both of the output of the CMOS image sensor and the synthesized event frame,

wherein the synthesized event frame is based on an output of a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data and an output of an inertial measurement unit over a time interval, and

wherein the semantic segmentation comprises a semantic label associated with a region of the time-stamped frame;

determining, based on the semantic segmentation, a simplified object representation in a coordinate space; and

updating a stable semantic map based on the simplified object representation.

9. The computer-implemented method of claim 8 , further comprising:

generating the semantic segmentation of the time-stamped frame by passing the output of the CMOS image sensor and the synthesized event frame through a deep convolutional neural network (CNN) to apply the semantic label to at least one pixel of the time-stamped frame.

10. The computer-implemented method of claim 8 , further comprising:

switching off, by a sensor scheduler, the CMOS image sensor based on a current value of one or more CMOS control factors.

11. The computer-implemented method of claim 8 , further comprising:

receiving, from a visual-inertial simultaneous location and mapping (SLAM) pipeline, pose data of the apparatus, the pose data associated with the time-stamped frame;

receiving, from the visual inertial SLAM pipeline, point cloud data associated with the time-stamped frame; and

determining the simplified object representation based on the pose data and the point cloud data,

wherein the simplified object representation comprises a higher-level geometric primitive associated with the semantic label.

12. The computer-implemented method of claim 11 , wherein the higher-level geometric primitive is a plane, and the computer-implemented method further comprises:

receiving gravity data associated with a three-dimensional environment represented in part by the point cloud data;

clustering the point cloud data in to a bounded planar region;

parameterizing the bounded planar region;

labeling the bounded planar region based on the semantic label; and

outputting a labeled plane.

13. The computer-implemented method of claim 11 , wherein the higher-level geometric primitive is a bounding box, and the computer-implemented method further comprises:

receiving gravity data associated with a three-dimensional environment represented in part by the point cloud data,

clustering the point cloud data associated with the semantic label;

determining a label for the clustered point cloud data;

determining coordinates of a minimum three-dimensional bounding box enclosing the clustered point cloud data; and

outputting a labeled bounding box.

14. The computer-implemented method of claim 8 , further comprising:

associating the simplified object representation with an element set of the stable semantic map;

determine a value of a confidence interval of the element set; and

responsive to the value of the confidence interval of the element set exceeding a threshold, update the stable semantic map by merging the simplified object representation into the element set.

15. A non-transitory computer-readable medium comprising program code, which, when executed by a processor, causes an apparatus to:

generate, a semantic segmentation of a time-stamped frame,

wherein the time-stamped frame is selectively based on: 1) one of an output of a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data or a synthesized event frame, or 2) both of the output of the CMOS image sensor and the synthesized event frame,

wherein the synthesized event frame is based on an output of a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data and an output of an inertial measurement unit over a time interval, and

wherein the semantic segmentation comprises a semantic label associated with a region of the time-stamped frame,

determine, based on the semantic segmentation, a simplified object representation in a coordinate space, and

update a stable semantic map based on the simplified object representation.

16. The non-transitory, computer-readable medium of claim 15 , further comprising program code, which, when executed by the processor, causes the apparatus to:

generate the semantic segmentation of the time-stamped frame by passing the output of the CMOS image sensor and the synthesized event frame through a deep convolutional neural network (CNN) to apply the semantic label to at least one pixel of the time-stamped frame.

17. The non-transitory, computer-readable medium of claim 15 , further comprising program code, which, when executed by the processor, causes the apparatus to:

switch off, by a sensor scheduler, the CMOS image sensor based on a current value of one or more CMOS control factors.

18. The non-transitory, computer-readable medium of claim 15 , further comprising program code, which, when executed by the processor, causes the apparatus to:

receive, from a visual-inertial simultaneous location and mapping (SLAM) pipeline, pose data of the apparatus, the pose data associated with the time-stamped frame,

receive, from the visual inertial SLAM pipeline, point cloud data associated with the time-stamped frame, and

determine the simplified object representation based on the pose data and the point cloud data,

wherein the simplified object representation comprises a higher-level geometric primitive associated with the semantic label.

19. The non-transitory, computer-readable medium of claim 18 , wherein the higher-level geometric primitive is a plane, and further comprising instructions, which, when executed by the processor, cause the apparatus to:

receive gravity data associated with a three-dimensional environment represented in part by the point cloud data,

cluster the point cloud data in to a bounded planar region,

parameterize the bounded planar region,

label the bounded planar region based on the semantic label, and

output a labeled plane.

20. The non-transitory, computer-readable medium of claim 18 , wherein the higher-level geometric primitive is a bounding box, and further comprising instructions, which, when executed by the processor, cause the apparatus to:

receive gravity data associated with a three-dimensional environment represented in part by the point cloud data,

cluster the point cloud data associated with the semantic label,

determine a label for the clustered point cloud data,

determine coordinates of a minimum three-dimensional bounding box enclosing the clustered point cloud data, and

output a labeled bounding box.