IP Library › Granted Patent US 12,282,331
Granted Patent B2
US 12,282,331 · App. 17/135,258 · Granted Apr 22, 2025

Mobile cleaning robot artificial intelligence for situational awareness

Inventors: Christopher V. Jones (Woburn, MA); Gary Ellis Hall (Malden, MA); Steven J. Baron (Nashua, NH); Brent Hild (Bedford, MA); Stefan Zickler (Bedford, MA); John Sinnigen (Cambridge, MA)
Assignee: iRobot Corporation
G05D1/0246A47L9/009A47L11/4011G05D1/0274G06F18/24143G06F18/2431G06T7/74G06V10/454G06V10/82G06V20/10G06V20/20G06V30/19173G06V40/10A47L2201/04G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,282,331
App. No.
17/135,258
Filed
Dec 28, 2020
Granted
Apr 22, 2025
Kind
B2
Art Unit
2665
USPC
382/153
Abstract

A mobile cleaning robot includes a cleaning head configured to clean a floor surface in an environment, and at least one camera having a field of view that extends above the floor surface. The at least one camera is configured to capture images that include portions of the environment above the floor surface. The robot includes a recognition module is configured to recognize objects in the environment based on the images captured by the at least one camera, in which the recognition module is trained at least in part using the images captured by the at least one camera. The robot includes a storage device is configured to store a map of the environment. The robot includes a control module configured to control the mobile cleaning robot to navigate in the environment using the map and operate the cleaning head to perform cleaning tasks taking into account of the objects recognized by the recognition module.

Claims (52)

1. A mobile cleaning robot comprising:

a cleaning head configured to clean a floor surface;

one or more sensors configured to obtain information about an environment, the one or more sensors comprising a microphone configured to capture audio signals from the environment;

a control module configured to:

communicate with a mobile computing device to cause the mobile computing device to display a message requesting a guided tour of the environment,

control the mobile cleaning robot to follow an individual through at least a portion of the environment responsive to an audible command from the individual, the audible command having been captured by the microphone, and

generate a map of the environment based at least in part on information obtained by the one or more sensors while following the individual through the environment; and

a storage device configured to store the generated map of the environment.

2. The mobile cleaning robot of claim 1 , wherein generating the map of the environment comprises labeling one or more rooms in the environment.

3. The mobile cleaning robot of claim 2 , wherein the control module is configured to determine a label for each of one or more rooms in the environment based on an audible signal from the individual, the audible signal having been captured by the microphone.

4. The mobile cleaning robot of claim 1 , wherein the one or more sensors comprise at least one camera configured to capture images that include regions of the environment above the floor surface.

5. The mobile cleaning robot of claim 4 , wherein the at least one camera is configured to capture the images as the mobile cleaning robot follows the individual through the at least a portion of the environment.

6. The mobile cleaning robot of claim 4 , wherein the control module is configured to control the mobile cleaning robot to return to one or more rooms of the environment to capture the images subsequent to controlling the mobile cleaning robot to follow the individual through the at least a portion of the environment.

7. The mobile cleaning robot of claim 4 , comprising a recognition module configured to determine a type of each of one or more objects in the environment based on the images captured by the at least one camera.

8. The mobile cleaning robot of claim 7 , wherein the recognition module is configured to determine the type of each of the one or more objects in the environment based at least in part on a type of room within which each of the one or more objects is located.

9. The mobile cleaning robot of claim 4 , wherein the captured images comprise images of a particular object in the environment, and wherein the at least one camera is configured to capture the images in response to the individual gesturing toward the particular object.

10. The mobile cleaning robot of claim 9 , wherein generating the map of the environment comprises labeling a type of the particular object, wherein the type of the particular object is determined based on an audible signal from the individual, the audible signal having been captured by the microphone.

11. The mobile cleaning robot of claim 4 , wherein the captured images comprise images of a particular object in the environment, and wherein the at least one camera of the mobile cleaning robot is configured to capture the images in response to the individual pointing a camera of the mobile computing device at the particular object.

12. A method performed by a mobile cleaning robot, the method comprising:

communicating with a mobile computing device to cause the mobile computing device to display a message requesting a guided tour of an environment,

capturing, by a microphone of the mobile cleaning robot, an audible command from an individual;

following the individual through at least a portion of the environment responsive to capturing the audible command;

obtaining, by one or more sensors of the mobile cleaning robot, information about the environment, the one or more sensors comprising the microphone; and

generating a map of the environment based at least in part on information obtained by the one or more sensors while following the individual through the environment.

13. The method of claim 12 , wherein generating the map of the environment comprises labeling one or more rooms in the environment.

14. The method of claim 13 , wherein labeling the one or more rooms in the environment comprises labeling the one or more rooms based on an audible signal from the individual captured by the microphone of the mobile cleaning robot.

15. The method of claim 12 , wherein obtaining information about the environment comprises capturing, by one or more cameras, images that include regions of the environment above a floor surface.

16. The method of claim 15 , comprising capturing the images while following the individual through the at least a portion of the environment.

17. The method of claim 15 , comprising recognizing that the individual is gesturing toward a particular object prior to capturing the images.

18. The method of claim 15 , comprising determining that the individual is pointing a camera of the mobile computing device at a particular object prior to capturing the images.

19. The method of claim 15 , comprising, subsequent to following the individual through the at least a portion of the environment, returning to one or more rooms of the environment to capture the images.

20. The method of claim 12 , wherein generating the map of the environment comprises determining a type of each of one or more objects in the environment.

21. The method of claim 20 , comprising determining the type of each of the one or more objects based on images captured by the one or more sensors.

22. The method of claim 20 , comprising determining the type of each of the one or more objects based, at least in part, on a type of room within which each of the one or more objects is located.

23. The method of claim 20 , wherein determining the type of each of the one or more objects is based on an audible signal from the individual captured by the microphone of the mobile cleaning robot.

24. A mobile cleaning robot comprising:

a cleaning head configured to clean a floor surface;

one or more sensors configured to obtain information about an environment;

a control module configured to:

communicate with a mobile computing device to cause the mobile computing device to display a message requesting a guided tour of the environment,

control the mobile cleaning robot to follow an individual through at least a portion of the environment responsive to a command from the individual, the command having been obtained by one or more of the sensors, and

generate a map of the environment based at least in part on information obtained by the one or more sensors while following the individual through the environment; and

a storage device configured to store the generated map of the environment.

25. The mobile cleaning robot of claim 24 , wherein generating the map of the environment comprises:

determining a label for each of one or more rooms in the environment based an audible signal from the individual, the audible signal having been captured by one or more of the sensors; and

labeling the one or more rooms in the map.

26. The mobile cleaning robot of claim 24 , wherein the one or more sensors comprise at least one camera configured to capture images including regions of the environment above the floor surface.

27. The mobile cleaning robot of claim 26 , wherein the camera is configured to capture the images as the mobile cleaning robot follows the individual through the at least a portion of the environment.

28. The mobile cleaning robot of claim 26 , wherein the control module is configured to control the mobile cleaning robot to return to an area of the environment to capture the images subsequent to controlling the mobile cleaning robot to follow the individual through the at least a portion of the environment.

29. The mobile cleaning robot of claim 26 , comprising a recognition module configured to determine a type of each of one or more objects in the environment based on the images captured by the at least one camera.

30. The mobile cleaning robot of claim 26 , wherein the captured images comprise images of a particular object in the environment, and wherein the at least one camera is configured to capture the images in response to the individual gesturing toward the particular object.

31. The mobile cleaning robot of claim 26 , wherein the captured images comprise images of a particular object in the environment, and wherein the at least one camera of the mobile cleaning robot is configured to capture the images in response to the individual pointing a camera of the mobile computing device at the particular object.

Assignments (5)
NOTICE OF ASSIGNMENT OF SECURITY INTEREST IN INTELLECTUAL PROPERTY Recorded Nov 25, 2025
From: TCG SENIOR FUNDING L.L.C., AS COLLATERAL AGENT
To: SANTRUM HONG KONG CO., LIMITED, AS SUCCESSOR COLLATERAL AGENT
Reel/Frame 073707/0516 →
SECURITY INTEREST Recorded Aug 9, 2023
From: IROBOT CORPORATION
To: TCG SENIOR FUNDING L.L.C., AS COLLATERAL AGENT
Reel/Frame 064532/0856 →
RELEASE OF SECURITY INTEREST Recorded Jul 31, 2023
From: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
To: IROBOT CORPORATION
Reel/Frame 064430/0001 →
SECURITY INTEREST Recorded Nov 3, 2022
From: IROBOT CORPORATION
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 061878/0097 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 16, 2021
From: JONES, CHRISTOPHER V.; HALL, GARY ELLIS; BARON, STEVEN J.; HILD, BRENT; ZICKLER, STEFAN; SINNIGEN, JOHN
To: IROBOT CORPORATION
Reel/Frame 055939/0680 →
Continuity (2)
Continuation 15863591 · Jan 5, 2018
Related Publication 20210224579A1 · Jul 22, 2021
References Cited (86)
US 7706917B1 · Chiappetta · 2010 [cited by applicant]
US 8438695B2 · Gilbert, Jr. · 2013 [cited by applicant]
US 9229454B1 · Chiappetta · 2016 [cited by applicant]
US 9380922B2 · Duffley · 2016 [cited by examiner]
US 10293489B1 · Cui · 2019 [cited by applicant]
US 10878294B2 · Jones et al. · 2020 [cited by applicant]
US 20070150094A1 · Huang · 2007 [cited by applicant]
US 20090281661A1 · Dooley · 2009 [cited by applicant]
US 20120182392A1 · Kearns · 2012 [cited by applicant]
US 20120185094A1 · Rosenstein · 2012 [cited by applicant]
US 20130204463A1 · Chiappetta · 2013 [cited by applicant]
US 20130206177A1 · Burlutskiy · 2013 [cited by applicant]
US 20130226344A1 · Wong · 2013 [cited by applicant]
US 20140070470A1 · Goldberg · 2014 [cited by examiner]
US 20140207282A1 · Angle · 2014 [cited by applicant]
US 20140336863A1 · So et al. · 2014 [cited by applicant]
US 20150362921A1 · Hanaoka · 2015 [cited by applicant]
US 20160274580A1 · Jung · 2016 [cited by applicant]
US 20160297072A1 · Williams · 2016 [cited by applicant]
US 20160334800A1 · Han · 2016 [cited by applicant]
US 20160375592A1 · Szatmary · 2016 [cited by applicant]
US 20160379092A1 · Kutliroff · 2016 [cited by examiner]
US 20170090456A1 · Mao · 2017 [cited by applicant]
US 20170185085A1 · Storfer et al. · 2017 [cited by applicant]
US 20170203446A1 · Dooley et al. · 2017 [cited by applicant]
US 20170312916A1 · Williams et al. · 2017 [cited by applicant]
US 20170355081A1 · Fisher et al. · 2017 [cited by applicant]
US 20170361468A1 · Cheuvront et al. · 2017 [cited by applicant]
US 20190061157A1 · Suvarna · 2019 [cited by examiner]
US 20190213438A1 · Jones et al. · 2019 [cited by applicant]
US 20200225673A1 · Ebrahimi Afrouzi · 2020 [cited by examiner]
CN 101101203 · 2008 [cited by applicant]
CN 201572040U · 2010 [cited by applicant]
CN 102083352A · 2011 [cited by applicant]
CN 104737085A · 2015 [cited by applicant]
CN 105395144 · 2016 [cited by applicant]
CN 205391066 · 2016 [cited by applicant]
CN 105892321 · 2016 [cited by applicant]
CN 107041718 · 2017 [cited by applicant]
CN 107256019 · 2017 [cited by applicant]
CN 107518833 · 2017 [cited by applicant]
DE 102015119501 · 2017 [cited by applicant]
EP 3575046 · 2019 [cited by applicant]
JP H08166818 · 1996 [cited by applicant]
JP 2005211359 · 2005 [cited by applicant]
JP 2005218578 · 2005 [cited by applicant]
JP 2005312893 · 2005 [cited by applicant]
JP 2006201829 · 2006 [cited by applicant]
JP 2007034561 · 2007 [cited by applicant]
JP 2013012200 · 2013 [cited by applicant]
JP 2013085958 · 2013 [cited by applicant]
JP 2013248065 · 2013 [cited by applicant]
JP 2014079513 · 2014 [cited by applicant]
JP 2015521771 · 2015 [cited by applicant]
JP 2016513981 · 2017 [cited by applicant]
JP 2017143901 · 2017 [cited by applicant]
KR 1020160048750 · 2016 [cited by applicant]
WO WO2017097893 · 2017 [cited by applicant]
WO WO2017112214 · 2017 [cited by applicant]
European Partial Search Report in European Appln. No. 22168033.3, mailed Oct. 11, 2022, 13 pages. [cited by applicant]
Redmon, Yolo9000, Better Faster, stronger, Jun. 8, 2015, 9, May 27, 2021. [cited by applicant]
Redmon, You Only Look Once: Unified, Real-Time Object Detection, Jun. 8, 2015, 10, May 27, 2021. [cited by applicant]
YOLO: Real-Time Object Detection, Nov. 29, 2016, 10, https://pjreddie.com/darknet/yolo/, May 27, 2021. [cited by applicant]
Angra et al., “Machine Learning and its Applications: A Review,” 2017 International Conference on Big Data Analytics and Computational Intelligence (ICBDAC), Mar. 23-25, 2017, 4 pages. [cited by applicant]
Bagnell et al., “Learning for Autonomous Navigation,” IEEE Robotics & Automation Magazine, Jun. 2010, 11 pages. [cited by applicant]
Ellis et al., “Autonomous Navigation and Sign Detector Learning,” 2013 IEEE Workshop on Robot Vision (WORV), Jan. 15-17, 2013, 8 pages. [cited by applicant]
European Search Report in European Application No. 18206656.3, dated Apr. 26, 2019, 9 pages. [cited by applicant]
European Search Report in European Appln. No. 18206656.3, dated Mar. 9, 2020, 4 pages. [cited by applicant]
Fromm et al., “Robust Multi-Algorithm Object Recognition Using Machine Learning Methods,” 2012 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) Sep. 13-15, 2012. Hamburg,… [cited by applicant]
Golding et al., “Indoor navigation using a diverse set of cheap, wearable sensors,” Digest of Papers. Third International Symposium on Wearable Computers, Oct. 18-19, 1999, 8 pages. [cited by applicant]
Howard et al., “ [cited by applicant]
LeCun et al., “Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting,” Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004,… [cited by applicant]
Louridas et al., “Machine Learning,” IEEE Software, vol. 33 Issue 5 ⋅ Sep.-Oct. 2016, 6 pages. [cited by applicant]
Malowany et al., “Visual Reinforcement Learning for Object Recognition in Robotics,” 2016 ICSEE International Conference on the Science of Electrical Engineering, Jan. 1995, 14(1):1-5. [cited by applicant]
Moore et al., “Efficient Memory-based Learning for Robot Control,” Dissertation submitted for the degree of Doctor of Philosophy in the University of Cambridge, Oct. 1990, 42 pages. [cited by applicant]
Murase et al., “Visual Learning and Recognition of 3-D Objects from Appearance,” International Journal of Computer Vision, 1995, 14:5-24. [cited by applicant]
Paletta et al., “Active object recognition by view integration and reinforcement learning,” Robotics and Autonomous Systems, 2000, 31:71-86. [cited by applicant]
Pontil et al. “Support Vector Machines for 3D Object Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Jun. 1998, 20(6):637-646. [cited by applicant]
Stauffer et al., “Learning Patterns of Activity Using Real-Time Tracking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Aug. 2000, 22(8):747-757. [cited by applicant]
Thrun, “Learning metric-topological maps for indoor mobile robot navigation,” Artificial Intelligence, 1998, 99:21-71. [cited by applicant]
Tsutsumi et al., “Hybrid Approach of Video Indexing and Machine Learning for Rapid Indexing and Highly Precise Object Recognition,” Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205), Oct… [cited by applicant]
Viola et al., “Rapid Object Detection using a Boosted Cascade of Simple Features,” Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Dec. 8-14, 2001, 8 pages. [cited by applicant]
Xu et al., “Extreme Learning Machine Based Fast Object Recognition,” 2012 15th International Conference on Information Fusion, Jul. 9-12, 2012, 7 pages. [cited by applicant]
Zuo et al., “A Reinforcement Learning Based Robotic Navigation System,” 2014 IEEE International Conference on Systems, Man, and Cybernetics, Oct. 5-8, 2014, San Diego, CA, USA, 6 pages. [cited by applicant]
Extended European Search Report in European Appln. No. 22168033.8, dated Jan. 12, 2023, 12 pages. [cited by applicant]
en.wikipedia.org [online], “Simultaneous localization and mapping,” Nov. 15, 2023, retrieved on Jan. 16, 2024, retrieved from URL<https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping>, 10 pages. [cited by applicant]
Cited By (1)
US 12,530,033