System and method of semantic segmentation for a cleaning device
A system and method of semantic segmentation for an autonomous or semi-autonomous cleaning device. By using machine learning and visual data, one can teach an algorithm (e.g., a deep neural network) to distinguish between ground and non-ground spaces. The non-ground spaces are passed to the cleaning device path planning to avoid these areas. The algorithm may be programmed to detect low obstacles (e.g., cardboard boxes, low pallets, forklift tines) and other low-height obstacles. Semantic segmentation obstacle avoidance adds another layer of safety to perception safety protocols of a cleaning device.
1 . A computer-implemented method for using semantic segmentation for identifying low-height obstacle avoidance of a semi-autonomous cleaning apparatus, the method comprising the steps of:
receiving an RGB image and a depth image from the cleaning apparatus and aligning and synchronizing the RGB image and the depth image;
generating a height image from the aligned depth image, each pixel of the height image indicating height relative to the floor;
running a trained neural network inference using at least the height image and the RGB image to obtain a binary classification image distinguishing floor from non-floor;
creating an obstacle point cloud from non-floor pixels in the binary classification image while excluding pixels occluded by a depth camera occlusion mask;
converting the obstacle point cloud to a laserscan and sending the laserscan to a cost map used by a planner to generate a path that avoids the obstacles; and
passing the planned path to the cleaning apparatus and executing the path to avoid the obstacles.
2 . The method of claim 1 wherein the step of receiving visual data further comprises receiving visual data from a camera or one or more sensors.
3 . The method of claim 1 wherein low-height obstacle comprise at least one of pallets, cardboard boxes and forklift tines.
4 . The method of claim 1 wherein the binary classification image identifies floor as a first class and non-floor as a second class.
5 . The method of claim 1 wherein generating the height image comprises producing pixel values mapped to heights in the range of approximately −0.5 m to 2.0 m relative to the floor.
6 . The method of claim 1 further comprising monitoring the semantic-segmentation pipeline and, when inference is unavailable on a machine-learning accelerator, pausing the cleaning apparatus while switching to CPU inference and resuming when healthy, and stopping the cleaning apparatus if both accelerator and CPU inference are unavailable.