Method and device for detecting electric vehicle using external camera and electric vehicle charging robot using the same
View Patent ↗The present disclosure provides a method of detecting a position of a vehicle such as an electric vehicle (EV) using a camera to determine whether the EV may be parked or located in a proper position. The vehicle position detection method, which may be performed by a device comprising a processor, includes: acquiring an original image of a vehicle from a camera; converting the original image into a bird's eye view image; detecting a bounding box associated with of the vehicle in the bird's eye view image by using the original image and the bird's eye view image; projecting the bird's-eye view image including the bounding box into a plane parallel with a ground to generate a projection image; and estimating a distance to the vehicle from a predetermined reference point or an orientation of the vehicle with respect to a predetermined reference line.
1 . A vehicle position detection method performed by a device comprising a processor, the method comprising:
acquiring a plurality of original images of a vehicle from a plurality of cameras;
converting each of the plurality of original images into a bird's eye view image;
detecting a bounding box associated with the vehicle in each of the bird's eye view images by using the original image and the bird's eye view image that correspond to each other;
projecting the bird's-eye view images including the bounding box into a plane parallel with a ground to generate a projection image; and
estimating a distance to the vehicle from a predetermined reference point or an orientation of the vehicle with respect to a predetermined reference line; and
controlling a manipulator to drive a robot arm when the orientation of the vehicle or the distance to the vehicle is within a predetermined range,
wherein projecting the bird's-eye view images including the bounding box into the plane parallel with the ground comprises:
mapping the bird's-eye view images corresponding to the plurality of original images along with respective bounding boxes to the projection image; and
calculating averages of the coordinates of a plurality of bounding boxes corresponding to respective bird's-eye view images to determine a final bounding box.
2 . The vehicle position detection method of claim 1 , wherein detecting the bounding box is performed by using an artificial neural network.
3 . The vehicle position detection method of claim 2 , wherein the artificial neural network comprises a backbone network configured to extract features in the plurality of original images and the bird's eye view images and a detection head configured to detect the bounding box based on the features.
4 . The vehicle position detection method of claim 2 , wherein detecting the bounding box comprises:
detecting the plurality of bounding boxes; and
excluding at least one bounding box among the plurality of bounding boxes by using a non-maxima suppression (NMS).
5 . The vehicle position detection method of claim 1 , wherein projecting the bird's-eye view images including the bounding box into the plane parallel with the ground comprises:
mapping the bird's-eye view images to the projection image based on the vertices of parking lot dividing lines.
6 . The vehicle position detection method of claim 5 , wherein estimating the distance to the vehicle from the predetermined reference point or the orientation of the vehicle with respect to the predetermined reference line comprises:
determining an angle formed by a line segment of the final bounding box and a corresponding one of the parking lot dividing lines as the orientation of the vehicle.
7 . The vehicle position detection method of claim 1 , wherein estimating the distance to the vehicle from the predetermined reference point or the orientation of the vehicle with respect to the predetermined reference line comprises:
determining a distance between a line segment of the final bounding box and the predetermined reference point in the projection image.
8 . A vehicle position detection device, comprising:
a memory having stored therein program instructions; and
a processor coupled to the memory and executing the program instructions stored in the memory,
wherein the processor is configured to:
acquire a plurality of original images of a vehicle from a plurality of cameras;
convert each of the plurality of original images into a bird's eye view image;
detect a bounding box associated with the vehicle in each of the bird's eye view images by using the original image and the bird's eye view image that correspond to each other,
project the bird's-eye view images including the bounding box into a plane parallel with a ground to generate a projection image, by:
mapping the bird's-eye view images corresponding to the plurality of original images along with respective bounding boxes to the projection image; and
calculating averages of the coordinates of a plurality of bounding boxes corresponding to respective bird's-eye view images to determine a final bounding box; and
estimate a distance to the vehicle from a predetermined reference point or an orientation of the vehicle with respect to a predetermined reference line; and
control a manipulator to drive a robot arm when the orientation of the vehicle or the distance to the vehicle is within a predetermined range.
9 . The vehicle position detection device of claim 8 , wherein the processor is further configured to detect the bounding box by implementing an artificial neural network.
10 . The vehicle position detection device of claim 9 , wherein the artificial neural network comprises a backbone network configured to extract features in the plurality of original images and the bird's eye view images and a detection head suitable for detecting the bounding box based on the features.
11 . The vehicle position detection device of claim 9 , wherein the processor is further configured to:
detect the plurality of bounding boxes; and
exclude at least one bounding box among the plurality of bounding boxes by using a non-maxima suppression (NMS).
12 . The vehicle position detection device of claim 8 , wherein the processor is further configured to:
map the bird's-eye view images to the projection image based on the vertices of parking lot dividing lines.
13 . The vehicle position detection device of claim 12 , wherein the processor is further configured to:
determine an angle formed by a line segment of the final bounding box and a corresponding one of the parking lot dividing lines as the orientation of the vehicle.
14 . The vehicle position detection device of claim 8 , wherein the processor is further configured to:
determine a distance between a line segment of the final bounding box and the predetermined reference point in the projection image.
15 . An electric vehicle charging robot, comprising:
a memory having program instructions stored therein;
a processor coupled to the memory and executing the program instructions stored in the memory;
a robot arm having a charging plug connected to a power source and installed at a distal end and configured to implement translational movement, rotational movement, or a combination of translational movement and rotational movement; and
a manipulator configured to drive the robot arm,
wherein the processor is configured to:
acquire a plurality of original images of a vehicle from a plurality of cameras;
convert each of the plurality of original images into a bird's eye view image;
detect a bounding box associated with the vehicle in each of the bird's eye view images by using the original image and the bird's eye view image that correspond to each other;
project the bird's-eye view images including the bounding box into a plane parallel with a ground to generate a projection image, by:
mapping the bird's-eye view images corresponding to the plurality of original images along with respective bounding boxes to the projection image; and
calculating averages of the coordinates of a plurality of bounding boxes corresponding to respective bird's-eye view images to determine a final bounding box;
estimate a distance to the vehicle from a predetermined reference point or an orientation of the vehicle with respect to a predetermined reference line; and
control the manipulator to drive the robot arm when the orientation of the vehicle or the distance to the vehicle is within a predetermined range.
16 . A system comprising the electric vehicle charging robot of claim 15 , wherein one or more of the plurality of cameras are provided outside the electric vehicle charging robot.
17 . The system of claim 16 , wherein the processor is further configured to:
determine the orientation of the vehicle; and
when the orientation of the vehicle is out of the predetermined range, generate an alarm for a misalignment.
18 . The system of claim 16 , wherein the processor is further configured to:
determine the distance to the vehicle; and
when the distance to the vehicle is out of the predetermined range, generate an alarm for a displacement.