Target positioning method and device, and unmanned aerial vehicle
The present invention relates to a target positioning method and device, and an unmanned aerial vehicle. The method includes: acquiring a target image through an image acquisition device; acquiring the position information of a target in the target image according to the target image; acquiring original point clouds of an environment in front of the unmanned aerial vehicle through a depth sensor; acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and acquiring the position information of the target in a three-dimensional space according to the point clouds corresponding to the target.
1. A target positioning method, applied to an unmanned aerial vehicle, the unmanned aerial vehicle comprising an image acquisition device and a depth sensor, and the method comprising:
acquiring a target image through the image acquisition device;
acquiring the position information of a target in the target image according to the target image;
acquiring original point clouds of an environment in front of the unmanned aerial vehicle through the depth sensor;
acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and
acquiring the position information of the target in a three-dimensional space according to the point clouds corresponding to the target;
wherein the position information of the target in the target image comprises minimum bounding box coordinates of the target in the target image, and acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image comprises:
projecting the original point clouds to a plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located; and
determining the original point clouds of which the coordinates fall into a minimum bounding box in the original point clouds as the point clouds corresponding to the target.
2. The target positioning method according to claim 1 , wherein projecting the original point clouds into the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located comprises:
transforming the coordinates of the original point clouds in a depth sensor coordinate system into the coordinates of the original point clouds in an image acquisition device coordinate system; and
projecting the coordinates of the original point clouds in the image acquisition device coordinate system to the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located.
3. The target positioning method according to claim 2 , wherein the coordinates of the original point clouds in the image acquisition device coordinate system satisfy:
(
x
′
y
′
z
′
)
=
T
×
(
x
y
z
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, (x, y, z) are the coordinates of the original point clouds in the depth sensor coordinate system, and T is a transformation matrix from the depth sensor coordinate system to the image acquisition device coordinate system.
4. The target positioning method according to claim 3 , wherein the transformation matrix T from the depth sensor coordinate system to the image acquisition device coordinate system satisfies:
T=T 2 T ×T 1 ,
wherein T 1 is a transformation matrix from the depth sensor coordinate system to an unmanned aerial vehicle fuselage coordinate system, and T 2 is a transformation matrix from the image acquisition device coordinate system to the unmanned aerial vehicle fuselage coordinate system.
5. The target positioning method according to claim 1 , wherein a coordinates (u, v) of the original point clouds in the plane where the target image is located satisfy:
d
×
(
u
v
)
=
k
×
(
x
′
y
′
z
′
)
and
k
=
(
f
x
0
c
x
0
f
y
c
y
0
0
1
)
and
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, d is the depth of the original point clouds, and k is an intrinsic matrix of the image acquisition device.
6. The target positioning method according to claim 1 , wherein before acquiring the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image, the method further comprises:
filtering the original point clouds of which the coordinates fall into the minimum bounding box so as to remove the point clouds which do not belong to the target.
7. The target positioning method according to claim 1 , wherein acquiring the position information of the target in the three-dimensional space according to the point clouds corresponding to the target comprises:
establishing a three-dimensional map of the environment in front of the unmanned aerial vehicle according to the original point clouds of the environment in front of the unmanned aerial vehicle; and
taking an average value of the three-dimensional coordinates of the point clouds corresponding to the target, wherein the average value is configured to represent the position of the target in the three-dimensional map.
8. The target positioning method according to claim 1 , wherein the depth sensor comprises at least one of a binocular camera, a structured light camera, a time of flight (TOF) camera, and a laser radar.
9. A target positioning device, applied to an unmanned aerial vehicle, the unmanned aerial vehicle comprising an image acquisition device and a depth sensor, and the target positioning device comprising:
a processor configured to acquire a target image collected by an image acquisition device and acquire the position information of a target in the target image according to the target image;
acquire original point clouds of an environment in front of the unmanned aerial vehicle collected through the depth sensor;
acquire the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and
acquire the position information of the target in a three-dimensional space according to the point clouds corresponding to the target;
project the original point clouds to a plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located; and
determine the original point clouds of which the coordinates fall into a minimum bounding box in the original point clouds as the point clouds corresponding to the target.
10. The device according to claim 9 , wherein the processor is further configured to
transform the coordinates of the original point clouds in a depth sensor coordinate system into the coordinates of the original point clouds in an image acquisition device coordinate system; and
project the coordinates of the original point clouds in the image acquisition device coordinate system to the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located.
11. The device according to claim 10 , wherein the coordinates of the original point clouds in the image acquisition device coordinate system satisfy:
(
x
′
y
′
z
′
)
=
T
×
(
x
y
z
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, (x, y, z) are the coordinates of the original point clouds in the depth sensor coordinate system, and T is a transformation matrix from the depth sensor coordinate system to the image acquisition device coordinate system.
12. The device according to claim 11 , wherein the transformation matrix T from the depth sensor coordinate system to the image acquisition device coordinate system satisfies:
T=T 2 T ×T 1 ,
wherein T 1 is a transformation matrix from the depth sensor coordinate system to an unmanned aerial vehicle fuselage coordinate system, and T 2 is a transformation matrix from the image acquisition device coordinate system to the unmanned aerial vehicle fuselage coordinate system.
13. The device according to claim 9 , wherein a coordinates (u, v) of the original point clouds in the plane where the target image is located satisfy:
d
×
(
u
v
)
=
k
×
(
x
′
y
′
z
′
)
and
k
=
(
f
x
0
c
x
0
f
y
c
y
0
0
1
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, d is the depth of the original point clouds, and k is an intrinsic matrix of the image acquisition device.
14. The device according to claim 9 , wherein the processor is further configured to
filter the original point clouds of which the coordinates fall into the minimum bounding box so as to remove the point clouds which do not belong to the target.
15. The device according to claim 9 , wherein the processor is further configured to
establish a three-dimensional map of the environment in front of the unmanned aerial vehicle according to the original point clouds of the environment in front of the unmanned aerial vehicle; and
take an average value of the three-dimensional coordinates of the point clouds corresponding to the target, wherein the average value is configured to represent the position of the target in the three-dimensional map.
16. The device according to claim 9 , wherein the depth sensor comprises at least one of a binocular camera, a structured light camera, a time of flight (TOF) camera, and a laser radar.
17. A unmanned aerial vehicle, comprising
a fuselage;
an arm, connected to the fuselage;
a power unit, arranged on the arm and configured to provide power required by the flight of the unmanned aerial vehicle;
a depth sensor, arranged on the fuselage and configured to acquire original point clouds of an environment in front of the unmanned aerial vehicle;
an image acquisition device, arranged on the fuselage and configured to acquire a target image and acquire the position information of a target in the target image according to the target image; and
a visual chip, arranged on the fuselage, both the depth sensor and the image acquisition device being in communication connection with the visual chip, and
the visual chip being configured to:
acquire the point clouds corresponding to the target in the original point clouds according to the original point clouds and the position information of the target in the target image; and
acquire the position information of the target in a three-dimensional space according to the point clouds corresponding to the target;
project the original point clouds to a plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located; and
determine the original point clouds of which the coordinates fall into a minimum bounding box in the original point clouds as the point clouds corresponding to the target.
18. The unmanned aerial vehicle according to claim 17 , wherein the visual chip is further configured to
transform the coordinates of the original point clouds in a depth sensor coordinate system into the coordinates of the original point clouds in an image acquisition device coordinate system; and
project the coordinates of the original point clouds in the image acquisition device coordinate system to the plane where the target image is located to acquire the coordinates of the original point clouds in the plane where the target image is located.
19. The unmanned aerial vehicle according to claim 18 , wherein the coordinates of the original point clouds in the image acquisition device coordinate system satisfy:
(
x
′
y
′
z
′
)
=
T
×
(
x
y
z
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, (x, y, z) are the coordinates of the original point clouds in the depth sensor coordinate system, and T is a transformation matrix from the depth sensor coordinate system to the image acquisition device coordinate system.
20. The unmanned aerial vehicle according to claim 19 , wherein the transformation matrix T from the depth sensor coordinate system to the image acquisition device coordinate system satisfies:
T=T 2 T ×T 1 ,
wherein T1 is the transformation matrix from the depth sensor coordinate system to a fuselage coordinate system, and T2 is the transformation matrix from the image acquisition device coordinate system to the fuselage coordinate system.
21. The unmanned aerial vehicle according to claim 17 , wherein a coordinates (u, v) of the original point clouds in the plane where the target image is located satisfy:
d
×
(
u
v
)
=
k
×
(
x
′
y
′
z
′
)
and
k
=
(
f
x
0
c
x
0
f
y
c
y
0
0
1
)
,
wherein (x′, y′, z′) are the coordinates of the original point clouds in the image acquisition device coordinate system, d is the depth of the original point clouds, and k is an intrinsic matrix of the image acquisition device.
22. The unmanned aerial vehicle according to claim 17 , wherein the visual chip is further configured to
filter the original point clouds of which the coordinates fall into the minimum bounding box so as to remove the point clouds which do not belong to the target.
23. The unmanned aerial vehicle according to claim 17 , wherein the visual chip is further configured to
establish a three-dimensional map of the environment in front of the unmanned aerial vehicle according to the original point clouds of the environment in front of the unmanned aerial vehicle; and
take an average value of the three-dimensional coordinates of the point clouds corresponding to the target, wherein the average value is configured to represent the position of the target in the three-dimensional map.
24. The unmanned aerial vehicle according to claim 17 , wherein the depth sensor comprises at least one of
a binocular camera, a structured light camera, a time of flight (TOF) camera, and a laser radar.