Systems and methods for capturing images and annotating the captured images with information

1. An autonomous mobile robot, comprising:

a drive configured to maneuver the autonomous mobile robot in an operating environment;

a camera having a field of view in a drive direction of the autonomous mobile robot;

at least one sensor configured to detect obstacles in the drive direction responsive to sensor events;

a frame buffer configured to store image frames obtained by the camera and corresponding to regions of the operating environment;

a memory device configured to store a learned data set comprising a plurality of descriptors corresponding to pixel patches of the image frames and one or more classifiers therefor; and

a controller configured to update one or more of the descriptors based on a sensor event indicative of one or more of the obstacles detected by the at least one sensor responsive to operation of the drive, and to operate the drive to maneuver the autonomous mobile robot to traverse one of the regions previously avoided or avoid one of the regions previously attempted to traverse responsive to a retraining process for the one or more classifiers based on the one or more of the descriptors that were updated.

2. The autonomous mobile robot of claim 1 , wherein the controller is further configured to identify ones of the image frames as corresponding to different views of a same region among the regions of the operating environment, and wherein the retraining process is performed responsive to inconsistency of respective outputs of the one or more classifiers of the learned data set for the ones of the image frames.

3. The autonomous mobile robot of claim 2 , wherein the controller is configured to execute the retraining process for the one or more classifiers of the learned data set based on a time of day.

4. The autonomous mobile robot of claim 3 , wherein the controller is further configured to track movement of one of the pixel patches through the ones of the image frames, wherein the one of the pixel patches comprises sunlight moving across a floor surface based on the time of day.

5. The autonomous mobile robot of claim 1 , wherein the controller is further configured to continuously update the descriptors of the learned data set based on subsequent image frames obtained by the camera, and execute the retraining process for the one or more classifiers of the learned data set responsive thereto.

6. The autonomous mobile robot of claim 5 , wherein the controller is configured to execute a training process for the one or more classifiers based on an initial data set prior to executing the retraining process, wherein the initial data set does not correspond to the regions of the operating environment.

7. The autonomous mobile robot of claim 1 , wherein the controller is configured to generate a map of the operating environment using visual simultaneous localization and mapping (VSLAM) based on the image frames, and is configured to annotate one or more of the pixel patches in the image frames as corresponding to the one or more of the obstacles such that the map is indicative of a presence of the one or more of the obstacles.

8. The autonomous mobile robot of claim 1 , wherein the camera comprises one or more cameras mounted on a top and/or bottom of the autonomous mobile robot.

9. The autonomous mobile robot of claim 1 , wherein the autonomous mobile robot further comprises an input/output interface that is configured to wirelessly communicate with at least one remote computing device that performs the retraining process.

10. The autonomous mobile robot of claim 1 , wherein the controller is further configured to:

track ones of the pixel patches of the image frames as corresponding to traversable floor based on the one or more of the descriptors thereof;

determine that the ones of the pixel patches correspond to non-traversable non-floor based on the sensor event detected at a corresponding region among the regions of the operating environment;

retrieve, from the frame buffer, an image frame among the image frames, wherein the image frame is captured immediately prior to the sensor event; and

update the one or more of the descriptors by generating a non-floor descriptor corresponding to the ones of the pixel patches in the image frame captured immediately prior to the sensor event, wherein the image frame captured immediately prior to the sensor event has a pose of the autonomous mobile robot associated therewith, and storing the non-floor descriptor in the learned data set.

11. The autonomous mobile robot of claim 10 , wherein the field of view includes a portion of a floor surface, and wherein the controller is further configured to:

generate the non-floor descriptor corresponding to characteristics of the non-traversable non-floor at a top of the image frame captured immediately prior to the sensor event;

generate a floor descriptor corresponding to characteristics of the floor surface at a bottom of the image frame captured immediately prior to the sensor event; and

store the floor descriptor and the non-floor descriptor in the learned data set.

12. The autonomous mobile robot of claim 1 , wherein the controller is further configured to:

track ones of pixel patches of the image frames as corresponding to non-traversable non-floor based on the one or more of the descriptors corresponding thereto;

determine that the ones of the pixel patches correspond to traversable floor based on absence of the sensor event at a corresponding region among the regions of the operating environment; and

update the one or more of the descriptors by generating a floor descriptor corresponding to the ones of the pixel patches in the image frames, the image frames having respective poses of the autonomous mobile robot associated therewith, and storing the floor descriptor in the learned data set.

13. The autonomous mobile robot of claim 1 , wherein the at least one sensor comprises an inertial measurement unit (IMU), an odometry sensor, a piezoelectric sensor, a structured light optical sensor, an infrared (IR) proximity sensor, a laser sensor, a lidar sensor, a volumetric point cloud sensor, a time of flight sensor, a PIXART imaging sensor, a PRIMESENSE sensor, and/or an RGBD sensor.

14. A method of operating an autonomous mobile robot, the method comprising:

maneuvering, by a drive system, the autonomous mobile robot in an operating environment;

obtaining, by a camera having a field of view in a drive direction of the autonomous mobile robot, a plurality of image frames corresponding to regions of the operating environment;

storing the image frames in a frame buffer;

detecting, by at least one sensor, one or more obstacles in the drive direction;

storing, in a memory device, a learned data set comprising a plurality of descriptors corresponding to pixel patches in the image frames;

updating, by a controller, one or more of the descriptors based on a sensor event indicative of the one or more of the obstacles detected by the at least one sensor responsive to the maneuvering; and

operating, by the controller, the drive to maneuver the autonomous mobile robot to traverse one of the regions previously avoided or avoid one of the regions previously attempted to traverse responsive to a retraining process for one or more classifiers based on the one or more of the descriptors that were updated.

15. The method of claim 14 , further comprising:

identifying, by the controller, ones of the image frames as corresponding to different views of a same region among the regions of the operating environment,

wherein the retraining process is performed responsive to inconsistency of respective outputs of the one or more classifiers of the learned data set for the ones of the image frames.

16. The method of claim 15 , further comprising:

executing, by the controller, the retraining process for the one or more classifiers of the learned data set based on a time of day.

17. The method of claim 16 , further comprising:

tracking, by the controller, movement of one of the pixel patches through the ones of the image frames, wherein the one of the pixel patches comprises sunlight moving across a floor surface based on the time of day.

18. The method of claim 14 , further comprising:

generating, by a controller, a map of the operating environment using visual simultaneous localization and mapping (VSLAM) based on the image frames; and

annotating, by the controller, one or more of the pixel patches in the image frames as corresponding to the one or more of the obstacles such that the map is indicative of a presence of the one or more of the obstacles.

19. The method of claim 14 , wherein the at least one sensor comprises an inertial measurement unit (IMU), an odometry sensor, a piezoelectric sensor, a structured light optical sensor, an infrared (IR) proximity sensor, a laser sensor, a lidar sensor, a volumetric point cloud sensor, a time of flight sensor, a PIXART imaging sensor, a PRIMESENSE sensor, and/or an RGBD sensor.