Granted Patent B2

US 12,330,821 · App. 17/672,668 · Granted Jun 17, 2025

Inventors: Lei Xu (Hong Kong, CN); Erkang Cheng (Hong Kong, CN); Tingting Zhu (Hong Kong, CN)

Assignee: Nullmax (Hong Kong) Limited

B64U10/13G06V10/82G08G1/0108G08G1/0125G08G1/0141G08G1/0145G08G1/096725G08G1/096766H04W4/40B64U2101/30

Abstract

Claims (89)

1. An autonomous driving system, comprising:

an unmanned aerial vehicle (UAV) in the air, wherein the UAV includes:

at least one UAV camera configured to collect first raw sensor information,

wherein the first raw sensor information comprises visual images and distances from ground objects to the UAV,

a UAV processor configured to convert the first raw sensor information to first ground traffic information in a three-dimensional (3D) coordinate system, and

a UAV communication module configured to transmit the collected first ground traffic information,

wherein the UAV communication module is further configured to transmit the collected first ground traffic information to one or more additional UAVs or one or more control centers; and

a land vehicle communicatively connected to the UAV in the air, wherein the land vehicle is configured to store the UAV, wherein the land vehicle is configured to charge the UAV, wherein the land vehicle is configured to release or launch the UAV, and wherein the land vehicle further comprises:

one or more vehicle sensors configured to collect second raw sensor information surrounding the land vehicle,

wherein the second raw sensor information comprises visual images and distances from surrounding objects to the land vehicle,

a land vehicle processor configured to convert the second raw sensor information to second ground traffic information in a two-dimensional (2D) coordinate system,

wherein the land vehicle processor is configured to determine a direction associated with the ground objects and the surrounding objects based on the images captured by the respective raw sensor information,

wherein the land vehicle processor is further configured to release the UAV to a position of following or leading the land vehicle,

wherein the land vehicle processor is further configured to release the UAV when the land vehicle is approaching an intersection and the UAV is configured to fly toward the intersection, leading the land vehicle,

wherein when the UAV is at the intersection, the sensors of the UAV are further configured to collect first ground traffic information comprising positions of crosswalks, land dividing lanes, curb sides, and other vehicles, and

a land vehicle communication module configured to receive the first ground traffic information from the UAV,

wherein the land vehicle processor is further configured to convert the first ground traffic information in the three-dimensional (3D) coordinate system to the two-dimensional (2D) coordinate system,

wherein the land vehicle processor is further configured to combine the first ground traffic information and the second ground traffic information to generate a world model, wherein the world model is a dataset that includes coordinates of one or more still objects, predicted trajectories of one or more moving objects, coordinates of one or more traffic signals, and one or more accessible areas, and

wherein the land vehicle processor is further configured to generate decisions for the land vehicle based on the world model.

2. The autonomous driving system of claim 1 , wherein the first ground traffic information is formatted in a coordinate system with a position of the UAV as a coordinate origin.

3. The autonomous driving system of claim 2 , wherein the first ground traffic information includes:

a position of the land vehicle in the coordinate system;

first position information that indicates locations of the one or more still objects on the ground in the coordinate system with the position of the UAV as the coordinate origin;

first motion information that indicates velocities of the one or more moving objects on the ground;

first predicted trajectories of the one or more moving objects on the ground;

first status information that indicates statuses of the one or more traffic signals; and

first area information that indicates the one or more accessible areas to the land vehicle from a perspective of the UAV.

4. The autonomous driving system of claim 3 , wherein the first position information of the one or more still objects is formed as one or more sets of coordinates in the coordinate system, one or more point clouds, one or more semantic segments, or features of the still objects extracted by a neural network.

5. The autonomous driving system of claim 1 , wherein the second ground traffic information is formatted in a coordinate system with a position of the land vehicle as a coordinate origin.

6. The autonomous driving system of claim 5 , wherein the second ground traffic information include:

second position information that indicates locations of the one or more still objects surrounding the land vehicle in the coordinate system with the position of the land vehicle as the coordinate origin;

second motion information that indicates velocities of the one or more moving objects surrounding the land vehicle;

second predicted trajectories of the one or more moving objects surrounding the land vehicle;

second status information that indicates statuses of the one or more traffic signals; and

second area information that indicates the one or more accessible areas to the land vehicle from a perspective of the land vehicle.

7. The autonomous driving system of claim 6 , wherein the second position information is formed as one or more sets of coordinates in the coordinate system, one or more point clouds, one or more semantic segments, or features of the still objects extracted by a neural network.

8. The autonomous driving system of claim 1 , wherein the land vehicle processor is configured to:

convert, by a transformation neural network running on the processor, the first ground traffic information from a first coordinate system with a position of the UAV as a coordinate origin to a second coordinate system with a position of the land vehicle as the coordinate origin,

convert, by the transformation neural network running on the processor, coordinates of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas and predicted trajectories of the one or more moving objects identified in the first coordinate system to coordinates in the second coordinate system; and

determine the coordinates of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas in the world model based on the converted coordinates and the second ground traffic information.

9. The autonomous driving system of claim 1 , wherein the land vehicle processor is configured to:

convert, by a transformation neural network running on the processor, semantic segments of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas identified in the first coordinate system to semantic segment in the second coordinate system; and

determine semantic segments of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas and the predicted trajectories of the one or more moving objects identified in the world model based on the converted semantic segment and the second ground traffic information.

10. The autonomous driving system of claim 1 , wherein the land vehicle processor is configured to:

convert, by a transformation neural network running on the processor, point clouds of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas identified in the first coordinate system to point clouds in the second coordinate system; and

determine point clouds of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas and predicted trajectories of the one or more moving objects identified in the world model based on the converted point clouds and the second ground traffic information.

11. A land vehicle in an autonomous driving system, comprising:

a land vehicle communication module configured to receive first ground traffic information surrounding the land vehicle, wherein the first ground traffic information is generated by an unmanned aerial vehicle (UAV) in the air;

wherein the land vehicle is configured to store the UAV, wherein the land vehicle is configured to charge the UAV, and wherein the land vehicle is configured to release or launch the UAV,

wherein the first ground traffic information comprises visual images and distances from ground objects to the UAV,

wherein the first ground traffic information is in a three-dimensional (3D) coordinate system,

one or more vehicle sensors configured to collect second ground traffic information surrounding the land vehicle;

wherein the second ground traffic information comprises visual images and distances from surrounding objects to the land vehicle,

wherein the second ground traffic information is in a two-dimensional (2D) coordinate system,

a land vehicle processor configured to combine the first ground traffic information and the second ground traffic information to generate a world model;

wherein the land vehicle processor is configured to determine a direction associated with the ground objects and the surrounding objects based on the images captured by the respective sensor information,

wherein the land vehicle processor is further configured to convert the first ground traffic information in the three-dimensional (3D) coordinate system to the two-dimensional (2D) coordinate system,

wherein the land vehicle processor is further configured to generate decisions for the land vehicle based on the world model,

wherein the land vehicle processor is further configured to release the UAV to a position of following or leading the land vehicle,

wherein the land vehicle processor is further configured to release the UAV when the land vehicle is approaching an intersection and the UAV is configured to fly toward the intersection, leading the land vehicle,

wherein when the UAV is at the intersection, the sensors of the UAV are further configured to collect first ground traffic information comprising positions of crosswalks, land dividing lanes, curb sides, and other vehicles, and

wherein the world model is a dataset that includes coordinates of one or more still objects, predicted trajectories of one or more moving objects, coordinates of one or more traffic signals, and one or more accessible areas.

12. The land vehicle of claim 11 , wherein the first ground traffic information is formatted in a coordinate system with a position of the UAV as a coordinate origin.

13. The land vehicle of claim 12 , wherein the first ground traffic information includes:

a position of the land vehicle in the coordinate system;

first position information that indicates locations of the one or more still objects on the ground in the coordinate system with the position of the UAV as the coordinate origin;

first motion information that indicates velocities of the one or more moving objects on the ground;

first predicted trajectories of the one or more moving objects on the ground;

first status information that indicates statuses of the one or more traffic signals; and

first area information that indicates the one or more accessible areas to the land vehicle from a perspective of the UAV.

14. The land vehicle of claim 13 , wherein the first position information of the one or more still objects is formed as one or more sets of coordinates in the coordinate system, one or more point clouds, one or more semantic segments, or features of the still objects extracted by a neural network.

15. The land vehicle of claim 11 , wherein the second ground traffic information is formatted in a coordinate system with a position of the land vehicle as a coordinate origin.

16. The land vehicle of claim 15 , wherein the second ground traffic information include:

second position information that indicates locations of the one or more still objects surrounding the land vehicle in the coordinate system with the position of the land vehicle as the coordinate origin;

second motion information that indicates velocities of the one or more moving objects surrounding the land vehicle;

second predicted trajectories of the one or more moving objects surrounding the land vehicle;

second status information that indicates statuses of the one or more traffic signals; and

second area information that indicates the one or more accessible areas to the land vehicle from a perspective of the land vehicle.

17. The land vehicle of claim 16 , wherein the second position information is formed as one or more sets of coordinates in the coordinate system, one or more point clouds, one or more semantic segments, or features of the still objects extracted by a neural network.

18. The land vehicle of claim 11 , wherein the land vehicle processor is configured to:

convert, by a transformation neural network running on the land vehicle processor, the first ground traffic information from a first coordinate system with a position of the UAV as a coordinate origin to a second coordinate system with a position of the land vehicle as the coordinate origin,

convert, by the transformation neural network running on the land vehicle processor, coordinates of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas and predicted trajectories of the one or more moving objects identified in the first coordinate system to coordinates in the second coordinate system; and

determine the coordinates of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas in the world model based on the converted coordinates and the second ground traffic information.

19. The land vehicle of claim 11 , wherein the land vehicle processor is configured to:

convert, by a transformation neural network running on the land vehicle processor, semantic segments of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas identified in the first coordinate system to semantic segment in the second coordinate system; and

determine semantic segments of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas and the predicted trajectories of the one or more moving objects identified in the world model based on the converted semantic segment and the second ground traffic information.

20. The land vehicle of claim 11 , wherein the land vehicle processor is configured to:

convert, by a transformation neural network running on the land vehicle processor, point clouds of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas identified in the first coordinate system to point clouds in the second coordinate system; and

determine point clouds of the one or more still objects, the one or more moving objects, the one or more traffic signals, and the one or more accessible areas and predicted trajectories of the one or more moving objects identified in the world model based on the converted point clouds and the second ground traffic information.

Assignments (1)

Continuity (2)

References Cited (23)

Cited By (3)

US 12,617,414 US 12,737,903 US 12,738,071