Granted Patent B2

US 12,743,091 · App. 18/957,425 · Granted Sep 22, 2026

Inventors: Abdulrahman Althobaiti (Thuwal, SA); Fadl Abdellatif (Thuwal, SA)

Assignee: Saudi Arabian Oil Company

G05D1/2247B64C39/024B64D45/04B64U20/87G05D1/224G05D1/46B64U2101/30B64U2201/20G05D2109/20

Abstract

Claims (75)

1 . A system, comprising:

an unmanned aerial vehicle (UAV), the UAV comprising one or more location and orientation sensors, a first camera device, and a second camera device;

a first computing device communicatively coupled to the UAV; and

a second computing device communicatively coupled to one or more of the first computing device and the UAV, wherein

the first computing device comprises a processor and a memory having stored thereon machine-readable instructions that cause the processor, when executed, to:

obtain localization data associated with the UAV in relation to an environment of the UAV from the one or more location and orientation sensors of the UAV;

execute a navigation process by:

obtaining a live video moving image from the first camera device of the UAV;

determining a waypoint associated with an inspection point for inspecting a structure;

generating a navigation display interface for display on a first display device, the navigation display interface comprising a plurality of navigation augmented reality (AR) display elements related to the determined waypoint superimposed over the live video moving image obtained from the first camera device of the UAV; and

transmitting the navigation display interface to the second computing device,

wherein the plurality of navigation AR display elements are generated based on the localization data obtained from the one or more location and orientation sensors of the UAV.

2 . The system of claim 1 , wherein the navigation display interface is transmitted from the first computing device to the second computing device via a video call connection.

3 . The system of claim 2 , wherein the video call connection comprises a 5G connection.

4 . The system of claim 1 , wherein the second computing device comprises:

a communication interface adapted to receive the navigation display interface from the first computing device;

a second display device adapted to display the navigation display interface; and

a user interface adapted to receive one or more user inputs to the navigation display interface,

wherein the one or more user inputs are transmitted to the first computing device via the communication interface.

5 . The system of claim 4 , wherein the one or more user inputs are displayed on the first display device.

6 . The system of claim 4 , wherein the one or more user inputs are displayed on the first display device.

7 . The system of claim 1 , wherein the memory stores additional machine-readable instructions that cause the processor, when executed, to:

execute a precision landing process when the UAV reaches the determined waypoint by:

obtaining a live video moving image from the second camera device of the UAV;

generating a precision landing display interface for display on the first display device, the precision landing display interface comprising a plurality of precision landing AR display elements related to a landing target associated with the determined waypoint superimposed over the live video moving image obtained from the second camera device of the UAV; and

transmitting the precision landing display interface to the second computing device

wherein the plurality of precision landing AR display elements are generated based on the localization data obtained from the one or more location and orientation sensors of the UAV.

8 . The system of claim 7 , wherein the second computing device comprises:

a communication interface adapted to receive the precision landing display interface from the first computing device;

a second display device adapted to display the precision landing display interface; and

a user interface adapted to receive one or more user inputs to the precision landing display interface,

wherein the one or more user inputs are transmitted to the first computing device via the communication interface.

9 . A method, comprising:

obtaining, at a first computing device from an unmanned aerial vehicle (UAV), localization data associated with the UAV in relation to an environment of the UAV from one or more location and orientation sensors of the UAV;

executing, at the first computing device, a navigation process by:

obtaining a live video moving image from a first camera device of the UAV;

determining a waypoint associated with an inspection point for inspecting a structure;

generating a navigation display interface for display on a first display device of the first computing device, the navigation display interface comprising a plurality of navigation augmented reality (AR) display elements related to the determined waypoint superimposed over the live video moving image obtained from the first camera device of the UAV; and

transmitting the navigation display interface to a second computing device,

wherein the plurality of navigation AR display elements are generated based on the localization data obtained from the one or more location and orientation sensors of the UAV.

10 . The method of claim 9 , wherein the navigation display interface is transmitted from the first computing device to the second computing device via a video call connection.

11 . The method of claim 10 , wherein the video call connection comprises a 5G connection.

12 . The method of claim 9 , further comprising:

receiving, at the second computing device via a communications interface, the navigation display interface from the first computing device;

displaying, on a second display device of the second computing device, the navigation display interface; and

receiving, at the second computing device via a user interface, one or more user inputs to the navigation display interface,

transmitting, to the first computing device via the communication interface, the one or more user inputs.

13 . The method of claim 12 , wherein the one or more user inputs are displayed on the first display device.

14 . The method of claim 9 , further comprising:

executing, at the first computing device, a precision landing process when the UAV reaches the determined waypoint by:

obtaining a live video moving image from a second camera device of the UAV;

generating a precision landing display interface for display on the first display device, the precision landing display interface comprising a plurality of precision landing AR display elements related to a landing target associated with the determined waypoint superimposed over the live video moving image obtained from the second camera device of the UAV; and

transmitting the precision landing display interface to the second computing device,

wherein the plurality of precision landing AR display elements are generated based on the localization data obtained from the one or more location and orientation sensors of the UAV.

15 . The method of claim 14 , further comprising:

receiving, at the second computing device via a communications interface, the precision landing display interface from the first computing device;

displaying, on a second display device of the second computing device, the precision landing display interface; and

receiving, at the second computing device via a user interface, one or more user inputs to the precision landing display interface,

transmitting, to the first computing device via the communication interface, the one or more user inputs.

16 . The method of claim 15 , wherein the one or more user inputs are displayed on the first display device.

17 . An unmanned aerial vehicle (UAV), comprising:

a controller;

a plurality of location and orientation sensors;

an imaging system comprising a navigation imaging camera, a precision landing imaging camera, and one or more depth or stereo cameras; and

a communication interface adapted to communicate with a computing device,

wherein the controller is adapted to:

operate the UAV in a navigation mode by:

determining a location of the UAV based at least in part on information captured using the one or more depth or stereo cameras and the plurality of location and orientation sensors;

relaying a first live video moving image from the navigation imaging camera to the computing device; and

transmitting navigation localization data corresponding to one or more augmented reality (AR) navigation display elements to the computing device, the navigation localization data being obtained using the one or more depth or stereo cameras and the plurality of location and orientation sensors; and

upon determining that the location of the UAV is proximate a waypoint associated with an inspection point for inspecting a structure, operate the UAV in a precision landing mode by:

relaying a second live video moving image from the precision landing imaging camera to the computing device;

transmitting precision landing localization data corresponding to one or more AR prevision landing display elements to the computing device, the navigation localization data being obtained using the one or more depth or stereo cameras and the plurality of location and orientation sensors.

18 . The UAV of claim 17 , wherein the navigation localization data comprises Visual Odometry (VO) or Visual Inertial Odometry (VIO) based on information captured by the imaging system in conjunction with data from the plurality of location and orientation sensors.

19 . The UAV of claim 17 , wherein the precision landing localization data comprises a three-dimensional (3D) point cloud from the plurality of location and orientation sensors for determining a landing target location.

Assignments (1)

Continuity (2)

References Cited (62)