Conversion of cleaning robot camera images to floorplan for user interaction
In one embodiment, a method is disclosed for a robot (e.g., cleaning robot) to produce a bird's eye view (planar) map by transforming from a robot camera view and stitching together images tagged by location. The planar map can then be presented on a user interface as a floorplan, showing the location of objects (such as rugs). A camera is mounted in the cleaning robot sufficiently high in the cleaning robot housing to provide an angled view downward at a floor. The camera's field of view is captured as an image, and a portion or segment of that image is cropped. The cropped segment is transformed from the camera perspective to a planar view (before or after cropping), and is combined with other images to form a map of a floorplan.
1. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:
providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;
creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;
creating a rectilinear grid of the floorplan using the localization data;
capturing a series of images of the floor with the camera at different locations as the floor-traversing cleaning robot traverses the floor;
determining a location of the floor-traversing cleaning robot for each of the images;
tagging each of the series of images with a location of the floor-traversing cleaning robot at a time of image capture;
cropping each of the images to identify at least one image segment in each of the images;
transforming the image segment from a robot camera perspective view to a planar view:
populating rectangles in the rectilinear grid of the floorplan with each of the image segments at the robot location in the planar view to provide an image populated floorplan, with an offset corresponding to a difference between the robot position and the position of the image segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;
displaying the image populated floorplan to a user; and
accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot;
wherein the field of view of the camera begins 5-25 centimeters in front of the floor-traversing cleaning robot.
2. The method of claim 1 further comprising:
providing a lens in front of the robot camera;
applying a distortion algorithm to the series of images to correct for an image distortion due to the lens.
3. The method of claim 1 further comprising:
detecting tilting of the floor-traversing cleaning robot beyond a first threshold; and
correcting the image pixel position or discarding an image.
4. The method of claim 1 further comprising:
capturing segments for an area that is the same size or larger than a minimum resolution of the floorplan.
5. The method of claim 1 , further comprising:
capturing overlapping segments; and
adjusting alignment adjustment by matching an overlapping portion of one segment with an overlapping portion of an adjacent segment.
6. The method of claim 1 further comprising:
providing a source of illumination on the floor-traversing cleaning robot for a field of view of the camera;
storing a baseline image brightness value for the source of illumination;
detecting variances in illumination of different segments from a baseline; and
making any needed corrections in the brightness of the segments due to illumination variances.
7. The system of claim 1 further comprising:
a LIDAR module mounted in the floor-traversing cleaning robot and configured to determine a location of the floor-traversing cleaning robot using a SLAM algorithm.
8. The method of claim 1 wherein the user input comprises the user indicating at least one of areas where the floor-traversing cleaning robot should not go and areas that require more intensive cleaning.
9. A method for providing a floorplan from images captured by a floor-traversing robot, comprising:
providing a camera mounted in the floor-traversing robot sufficiently high in a robot housing to provide an angled view downward at a floor;
capturing a series of images of the floor with the camera at different locations as the floor-traversing robot traverses the floor;
determining a location of the floor-traversing robot for each of the images;
tagging each of the series of images with a location of the floor-traversing robot at a time of image capture;
cropping each of the images to identify at least one segment in each of the images;
transforming the segment from a robot camera perspective view to a planar view;
populating the floorplan with each of the segments at the robot location in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;
displaying the floorplan to a user;
accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing robot;
capturing the same segment multiple times to provide multiple versions of the same segment;
determining a weighted quality of each version of the same segment, with the weighed quality including at least one of the speed of the robot when an image was captured, vibration, angle (tilt), agreement with overlapping portions of other segments, and closeness of the segment to the camera and illumination; and
replacing a lower quality version with a higher quality version.
10. The method of claim 9 , wherein the weighted quality is determined using at least one of a speed of the robot when an image was captured, vibration, tilt of the robot, agreement with overlapping portions of other segments, closeness of the segment to the camera and illumination.
11. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:
providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;
creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;
creating a rectilinear grid of the floorplan using the localization data;
capturing a series of images of the floor with the camera;
providing a lens in front of the robot camera;
applying a distortion algorithm to the series of images to correct for an image distortion due to the lens;
determining a location of the floor-traversing cleaning robot for each of the images;
tagging each of the series of images with the location of the floor-traversing cleaning robot at a time of image capture;
detecting tilting of the cleaning robot beyond a first threshold;
correcting an image pixel position or discarding the image;
cropping each of the images to identify at least one segment in each of the images;
capturing segments for an area that is the same size or larger than a minimum resolution of the floorplan;
transforming the segment from a robot camera perspective view to a planar view;
populating rectangles in the rectilinear grid of the floorplan with each of the segments at the robot location in the planar view to provide an image populated floorplan by utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan, with an offset corresponding to a difference between the robot position and the position of the segment on the floor;
capturing overlapping segments;
adjusting alignment adjustment by matching an overlapping portion of one segment with an overlapping portion of an adjacent segment;
displaying the image populated floorplan to a user; and
accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot.
12. A method for providing a floorplan from images captured by a floor-traversing robot, comprising:
providing a camera mounted in the floor-traversing robot sufficiently high in a robot housing to provide an angled view downward at a floor;
capturing a series of images of the floor with the camera at different locations as the floor-traversing robot traverses the floor;
determining a location of the floor-traversing robot for each of the images;
synchronizing a frame rate of image capture with a rate of determining a location of the robot;
tagging each of the series of images with a location of the floor-traversing robot at a time of image capture;
cropping each of the images to identify at least one segment in each of the images;
transforming the segment from a robot camera perspective view to a planar view;
populating the floorplan with each of the segments at the robot location in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;
displaying the floorplan to a user; and
accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing robot.
13. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:
providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;
creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;
creating a rectilinear grid of the floorplan using the localization data;
capturing a series of images of the floor with the camera at different locations as the floor-traversing cleaning robot traverses the floor;
determining a location of the floor-traversing cleaning robot for each of the images;
tagging each of the series of images with a location of the floor-traversing cleaning robot at a time of image capture;
cropping each of the images to identify at least one image segment in each of the images;
transforming the image segment from a robot camera perspective view to a planar view;
populating rectangles in the rectilinear grid of the floorplan with each of the image segments at the robot location in the planar view to provide an image populated floorplan, with an offset corresponding to a difference between the robot position and the position of the image segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;
displaying the image populated floorplan to a user; and
accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot;
wherein the resolution of the camera is less than 1280×720 pixels.
14. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:
providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;
creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;
creating a rectilinear grid of the floorplan using the localization data;
capturing a series of images of the floor with the camera at different locations as the floor-traversing cleaning robot traverses the floor;
determining a location of the floor-traversing cleaning robot for each of the images;
tagging each of the series of images with a location of the floor-traversing cleaning robot at a time of image capture;
cropping each of the images to identify at least one image segment in each of the images;
transforming the image segment from a robot camera perspective view to a planar view:
populating rectangles in the rectilinear grid of the floorplan with each of the image segments at the robot location in the planar view to provide an image populated floorplan, with an offset corresponding to a difference between the robot position and the position of the image segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;
displaying the image populated floorplan to a user; and
accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot;
wherein the field of view of the camera is between 100 and 180 degrees and the camera is mounted 5-20 centimeters above the floor.
15. The system of claim 14 further comprising:
a lens mounted in front of the robot camera; and
non-transitory, computer-readable code having instructions for
applying a distortion algorithm to the series of images to correct for an image distortion due to the lens.
16. The system of claim 14 further comprising:
a tilt sensor mounted in the floor-traversing cleaning robot; and
non-transitory, computer-readable code imbedded in computer readable media having instructions for
detecting tilting of the floor-traversing cleaning robot beyond a first threshold; and
correcting the image pixel position or discarding an image.
17. The system of claim 14 further comprising:
a second camera mounted in the floor-traversing cleaning robot sufficiently high in the robot housing to provide an angled view downward at a floor with a field of view that overlaps with the first-mentioned camera.
18. The system of claim 14 further comprising:
a light mounted in the floor-traversing cleaning robot to provide illumination of a field of view of the camera;
non-transitory, computer-readable code imbedded in computer readable media having instructions for
storing a baseline image brightness value for a source of illumination,
detecting variances in illumination of different segments from a baseline; and
making any needed corrections in the brightness of the segments due to illumination variances.