Method for locating a warehousing robot, a method of constructing a map, robot and storage medium
Disclosed are a method for locating a warehousing robot, a method for constructing a map, a robot and a storage medium. In a specific embodiment, a semantic map of a warehouse environment is constructed in advance, and the semantic map comprises a plurality of objects existing in the warehouse environment and semantic information of the objects. In the localization process, a warehousing robot uses its own image sensor to acquire an image or video data of a surrounding environment ( 11 ), identifies target objects in the image or video data and semantic information of the target objects ( 12 ) to obtain the relative position relationship between each target object and the warehousing robot ( 13 ), and then determines the location of the warehousing robot in the semantic map based on the relative position relationship and the semantic information of each target object ( 14 ). The method for constructing a map is based on visual semantic localization. Because the method directly detects specific targets, the detection speed is fast, semantic information is rich, and the method is not easily influenced by other interference factors. The method gets rid of the dependence on signs in the warehouse environment and has high localization flexibility.
1 . A method for locating a warehousing robot, comprising:
acquiring an image or video data of a surrounding warehouse environment by using an image sensor on a warehousing robot during movement of the warehousing robot;
performing target detection and semantic recognition on the image or video data of the surrounding warehouse environment to obtain target objects in the surrounding warehouse environment and semantic information of the target objects;
calculating a relative position relationship between each target object and the warehousing robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system; and
determining a location of the warehousing robot in a semantic map according to the relative position relationship between each target object and the warehousing robot and the semantic information of each target object, wherein the semantic map comprises a plurality of objects in the warehouse environment and semantic information of the objects.
2 . The method according to claim 1 , wherein calculating the relative position relationship between each target object and the warehousing robot based on the transformation relationship between a sensor coordinate system and a robot coordinate system comprises:
calculating three-dimensional coordinates of pixels on the target object in the sensor coordinate system;
transforming the three-dimensional coordinates of the pixels on the target object from the sensor coordinate system to the robot coordinate system based on the transformation relationship between the sensor coordinate system and the robot coordinate system; and
calculating the relative position relationship between the target object and the warehousing robot according to the three-dimensional coordinates of the pixels on the target object in the robot coordinate system.
3 . The method according to claim 2 , wherein
the image sensor is a monocular camera and the target object is an object intersecting a ground plane or an object located on the ground plane, and calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system comprises: calculating the three-dimensional coordinates, on the ground plane, of the pixels intersecting the ground plane on the target object in combination with a photographic geometry of the monocular camera; or
the image sensor is a binocular camera, and calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system comprises: calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system by using a binocular stereo matching algorithm; or
the image sensor is an RGBD camera, and calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system comprises: calculating the three-dimensional coordinates of the pixels on the target object in the sensor coordinate system according to the matching relationship between an acquired RGB image containing the target object and a depth image containing the target object.
4 . The method according to claim 1 , wherein determining the location of the warehousing robot in a semantic map according to the relative position relationship between each target object and the warehousing robot and the semantic information of each target object comprises:
determining a location of the target object in the semantic map according to the semantic information of the target object; and
determining the location of the warehousing robot in the semantic map according to the relative position relationship between the target object and the warehousing robot and the location of the target object in the semantic map.
5 . The method according to claim 4 , wherein determining the location of the target object in the semantic map according to the semantic information of the target object comprises:
calculating an initial location of the warehousing robot in the semantic map based on other sensor data on the warehousing robot; and
searching for the location of the target object around the initial location in the semantic map according to the semantic information of the target object.
6 . The method according to claim 5 , wherein calculating the initial location of the warehousing robot in the semantic map based on other sensor data on the warehousing robot comprises:
calculating the initial location of the warehousing robot in the semantic map based on at least one of laser sensor data, IMU data and odometer data on the warehousing robot.
7 . The method according to claim 1 , further comprising:
acquiring the image or video data of the warehouse environment by using the image sensor on the warehousing robot during process of the warehousing robot traversing the warehouse environment;
performing target detection and semantic recognition on the image or video data of the warehouse environment to obtain a plurality of objects existing in the warehouse environment and semantic information of the objects;
calculating relative position relationships between the plurality of objects and the warehousing robot respectively based on the transformation relationship between the sensor coordinate system and the robot coordinate system; and
adding the semantic information of the plurality of objects to a basic environmental map according to the relative position relationships between the plurality of objects and the warehousing robot and a location of the warehousing robot in the basic environmental map when acquiring the image or video data of the warehouse environment, so as to obtain the semantic map of the warehouse environment.
8 . The method according to claim 7 , wherein when acquiring the image or video data of the warehouse environment, the method further comprises:
constructing the basic environmental map of the warehouse environment by using the SLAM technology, and determining the location of the warehousing robot in the basic environmental map when acquiring the image or video data of the warehouse environment.
9 . The method according to claim 1 , wherein the target objects at least comprise: shelves, intersecting lines between the shelves and a ground plane, and intersections formed by the shelves.
10 . The method according to claim 9 , wherein the target objects further comprise: at least one of shelf bases, fire hydrants and escape signs.
11 . A method for constructing a semantic map, comprising:
acquiring an image or video data of a warehouse environment by using an image sensor on a warehousing robot during process of the warehousing robot traversing the warehouse environment;
performing target detection and semantic recognition on the image or video data of the warehouse environment to obtain a plurality of objects existing in the warehouse environment and semantic information of the objects;
calculating relative position relationships between the plurality of objects and the warehousing robot respectively based on a transformation relationship between a sensor coordinate system and a robot coordinate system; and
adding the semantic information of the plurality of objects to a basic environmental map according to the relative position relationships between the plurality of objects and the warehousing robot and a location of the warehousing robot in the basic environmental map when acquiring the image or video data of the warehouse environment, so as to obtain the semantic map of the warehouse environment.
12 . A warehousing robot, comprising a device body including:
a memory for storing computer programs; and
a processor coupled to the memory and used for executing the computer programs to: acquire an image or video data of a surrounding warehouse environment by using an image sensor on a warehousing robot during movement of the warehousing robot; perform target detection and semantic recognition on the image or video data of the surrounding warehouse environment to obtain target objects in the surrounding warehouse environment and semantic information of the target objects; calculate a relative position relationship between each target object and the warehousing robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system; and determine a location of the warehousing robot in a semantic map according to the relative position relationship between each target object and the warehousing robot and the semantic information of each target object, wherein the semantic map comprises a plurality of objects in the warehouse environment and semantic information of the objects.
13 . The warehousing robot according to claim 12 , wherein the processor, when calculating the relative position relationship between each target object and the warehousing robot based on the transformation relationship between a sensor coordinate system and a robot coordinate system, is specifically configured to:
calculate three-dimensional coordinates of pixels on the target object in the sensor coordinate system;
transform the three-dimensional coordinates of the pixels on the target object from the sensor coordinate system to the robot coordinate system based on the transformation relationship between the sensor coordinate system and the robot coordinate system; and
calculate the relative position relationship between the target object and the warehousing robot according to the three-dimensional coordinates of the pixels on the target object in the robot coordinate system.
14 . The warehousing robot according to claim 12 , wherein the processor, when determining the location of the warehousing robot in a semantic map according to the relative position relationship between each target object and the warehousing robot and the semantic information of each target object, is specifically configured to:
determine a location of the target object in the semantic map according to the semantic information of the target object; and
determine the location of the warehousing robot in the semantic map according to the relative position relationship between the target object and the warehousing robot and the location of the target object in the semantic map.
15 . The warehousing robot according to claim 12 , wherein the processor is further configured to:
acquire the image or video data of the warehouse environment by using the image sensor on the warehousing robot during process of the warehousing robot traversing the warehouse environment;
perform target detection and semantic recognition on the image or video data of the warehouse environment to obtain a plurality of objects existing in the warehouse environment and semantic information of the objects;
calculate relative position relationships between the plurality of objects and the warehousing robot respectively based on the transformation relationship between the sensor coordinate system and the robot coordinate system; and
add the semantic information of the plurality of objects to a basic environmental map according to the relative position relationships between the plurality of objects and the warehousing robot and a location of the warehousing robot in the basic environmental map when acquiring the image or video data of the warehouse environment, so as to obtain a semantic map of the warehouse environment.