Granted Patent B1

US 11,726,490 · App. 17/127,849 · Granted Aug 15, 2023

Inventors: Ali Ebrahimi Afrouzi (San Diego, CA); Lukas Fath (York, CA); Chen Zhang (Richmond, CA); Brian Highfill (Castro Valley, CA)

Assignee: AI Incorporated

G05D1/0246G06T7/521G06T7/68G06T7/70

Abstract

Claims (74)

1. A robotic device, comprising:

a chassis including a set of wheels;

a motor to drive the set of wheels;

a battery to power the robotic device;

a controller in communication with the motor and wheels, the controller including a processor operable to control the motor and wheels to steer movement of the robotic device;

a plurality of sensors;

a camera; and,

a tangible, non-transitory, machine readable medium storing instructions that when executed by a processor effectuates operations comprising:

capturing, with the camera, one or more images of an environment of the robotic device as the robotic device drives back and forth in straight lines;

capturing, with the plurality of sensors, sensor data of the environment as the robotic device drives back and forth in the straight lines;

generating or updating, with the processor, a map of the environment based on at least one of the one or more images and the sensor data;

identifying, with the processor, one or more rooms in the map based on at least one of the one or more images and the sensor data;

receiving, with the processor, one or more multidimensional arrays including at least one parameter that is used to identify a feature included in the one or more images, wherein values of the at least one parameter correspond with pixels of a corresponding one or more images of the feature;

determining, with the processor, a position and orientation of the robotic device relative to the feature based on at least one of the one or more images and the sensor data; and

transmitting, with the processor, a signal to the processor of the controller to adjust a heading of the robotic device based on the position and orientation of the robotic device relative to the feature identified.

2. The robotic device of claim 1 , further comprising:

detecting, with the processor, one or more objects based on at least one of the one or more images and the sensor data;

determining, with the processor, a type of the one or more object based on at least one of the one or more images and the sensor data;

marking, with the processor, a location of the one or more objects in the map; and

determining or adjusting, with the processor, a movement path of the robotic device based on the location of the one or more objects.

3. The robotic device of claim 2 , further comprising:

determining, with the processor, areas of the environment that are high object density areas based on the location of the one or more objects; and

determining or adjusting, with the processor, the movement path of the robotic device based on the areas with high object density.

4. The robotic device of claim 1 , further comprising:

detecting, with the processor, one or more humans based on at least one of the one or more images and the sensor data;

recognizing, with the processor, the one or more humans based on at least one of the one or more images and the sensor data;

recording, with the processor, at least one of a day on which the one or more humans are detected, time, on which the one or more humans are detected, and a location of the one or more humans detected; and

planning, with the processor, one or more work sessions of the robotic device based on at least one of the day, the time, and the location.

5. The robotic device of claim 4 , wherein the processor plans the one or more work sessions in areas of the environment on days and times the areas of the environment are less likely to be occupied by the one or more humans.

6. The robotic device of claim 1 , further comprising:

recognizing, with the processor, the previously visited one or more rooms in the map based on at least one of the one or more images and the sensor data; and

determining, with the processor, to skip operating in the one or more rooms at least when the robotic device has already performed work in the one or more rooms.

7. The robotic device of claim 1 , further comprising:

determining or adjusting, with the processor, at least one of a schedule, a movement path, and a cleaning intensity of the robotic device based on at least one of the one or more images and the sensor data.

8. The robotic device of claim 7 , wherein the cleaning intensity comprises a cleaning intensity of at least one of a vacuum, a mop, and a UV light.

9. The robotic device of claim 1 , wherein the sensor data comprises data indicative of at least one of a floor type, debris accumulation, particulate density, friction between the set of wheels and a driving surface, friction between a brush of the robotic device and the driving surface, a location of one or more cliffs, a location of the robotic device, a location of the one or more rooms, a location of one or more objects, an object type of the one or more objects, a location in which one or more humans are detected, a day and time the one or more humans are detected, and facial recognition of the one or more humans.

10. The robotic device of claim 1 , further comprising:

generating, with the processor, simulated representations of the environment corresponding with possible locations of the robotic device; and

determining, with the processor, a most feasible location of the robotic device by comparing at least one of the one or more images and the sensor data against the simulated representations of the environment.

11. The robotic device of claim 1 , further comprising:

determining, with the processor, a probability of each location of the environment being occupied by an object based on at least one of the one or more images and the sensor data;

increasing, with the processor, a probability of a location being occupied by an object each time an object is detected in the location; and

marking, with the processor, the location as being occupied by an object in the map when the probability of the location being occupied by an object is greater than a predetermined amount.

12. The robotic device of claim 1 , wherein the robotic device continues to discover new areas while capturing new images and sensor data and the processor integrates the new images and sensor data into the map until the map is a closed loop.

13. The robotic device of claim 1 , further comprising:

determining, with the processor, a distance to one or more objects based on at least one of the one or more images and the sensor data.

14. The robotic device of claim 1 , wherein a movement path of the robotic device is perpendicular or parallel to a surface of the feature.

15. The robotic device of claim 1 , wherein the feature is a laser light projected onto a surface perpendicular or parallel to a movement path of the robotic device using one or more light emitters.

16. The robotic device of claim 1 , wherein the feature comprises a line, a curve, a polygon, one or more points, an edge, or a corner.

17. The robotic device of claim 1 , further comprising:

determining, with the processor, an amount of distortion of the feature in the one or more images based on at least a portion of the values of the at least one parameter; and

adjusting, with the processor of the controller, the heading of the robotic device by an amount proportional to an amount of distortion of the feature.

18. The robotic device of claim 17 , wherein determining the amount of distortion comprises at least one of:

counting a number of columns or rows of pixels found between a division line of an image of the one or more images and a first column or row of pixels containing at least one pixel with brightness intensity above a predetermined threshold in a first and second direction from the division line, and

subtracting a number of pixels from the other, the second direction being opposite the first, wherein a direction of the heading adjustment is indicated by a sign of the amount of distortion, wherein a positive and a negative amount of distortion indicate opposite directions; or

comparing the overlap of pixels between two images captured consecutively; or

identifying a position of a first point of the projected feature in an image of the one or more images by a first set of coordinates and a position of a second point of the projected feature in the image by a second set of coordinates, the second point symmetrically corresponding with the first point when the heading is accurate, and

determining the distance of the first point and second point from a division line in the image and subtracting one distance from the other.

19. A method for guiding a robotic device, comprising:

capturing, with a camera of the robotic device, one or more images of an environment of the robotic device as the robotic device drives within the environment, wherein a field of view of the camera captures a portion of the environment at a height equal to or greater than a height of the robotic device;

capturing, with a plurality of sensors of the robotic device, sensor data of the environment as the robotic device drives within the environment;

generating or updating, with a processor of the robotic device, a planar floor map of the environment based on at least some of the sensor data;

identifying, with the processor, one or more rooms based on an arrangement of areas in the planar floor map;

receiving, with the processor, at least one parameter that is used to identify a feature included in the one or more images to avoid accumulation of error in the planar floor map;

determining, with the processor, a position and orientation of the robotic device relative to the planar floor map based on at least one of the one or more images and the sensor data; and

transmitting, with the processor, a signal to a processor of a controller of the robotic device to adjust a heading of the robotic device based on the position and orientation of the robotic device relative to the planar floor map.

20. A tangible, non-transitory, machine readable medium storing instructions that when executed by a processor effectuates operations comprising:

capturing, with a camera of the robotic device, one or more image streams of an environment of the robotic device as the robotic device drives within an environment of the robotic device;

capturing, with a plurality of sensors of the robotic device, sensor data of the environment as the robotic device drives within the environment, wherein the plurality of sensors comprises at least one of an inertial measurement unit, a wheel odometer, and an optical tracking sensor;

generating or updating, with the processor of the robotic device, a map of the environment based on at least one of the one or more image streams and the sensor data while concurrently determining, with the processor, a position of the robotic device relative to the environment, by at least:

identifying features in the one or more image streams, wherein the features comprise at least one of a line, a curve, a polygon, a point, an edge, and a corner; and

tracking a correspondence between features in consecutive images in the one or more image streams and a change in the features observed in the consecutive images as the robotic device drives within the environment;

determining, with the processor, a change in a heading of the robotic device based on at least one the one or more image streams; and

transmitting, with the processor, a signal to a processor of a controller of the robotic device to adjust a heading of the robotic device towards a direction of a next desired location.

Continuity (16)

Cited By (1)

US 12,332,386