Granted Patent B2

US 12,729,973 · App. 17/675,472 · Granted Sep 8, 2026

Inventors: Rowoon An (Seoul, KR); Jae Yul Woo (Seoul, KR); Seunghyun Woo (Seoul, KR); Soobin Kim (Seoul, KR)

Assignees: Hyundai Motor Company; Kia Corporation

G01C21/3635G01C21/3807G01C21/3837G06F3/14G06T11/00G06T15/00G06T19/006

Abstract

Claims (57)

1 . A user device, comprising:

an image sensor configured to photograph a surrounding image;

a display module;

a communication circuitry configured to communicate with a server;

a database configured to store visual positioning system (VPS) map information based on augmented reality (AR) and two dimensional (2D) map information; and

a controller configured to:

identify a location of the user device based on the surrounding image and the VPS map information by using a VPS positioning model, the VPS positioning model being trained using metadata including the surrounding image and photographing direction information and excluding image data photographed in a direction of a ground;

generate 2D route information based on the 2D map information and delivery information received from the server;

generate three dimensional (3D) route information based on the delivery information and the VPS map information; and

control the display module to display the 2D route information and the 3D route information based on an actual distance between the location of the user device and a location of a point of interest (POI) included in the delivery information, the 3D route information being based on augmented reality;

wherein the controller is configured to:

based on the actual distance between the location of the user device and the location of the POI being greater than a predetermined distance, control the display module to display the 2D route information, and

based on the actual distance between the location of the user device and the location of the POI being less than or equal to the predetermined distance, control the display module to display the 3D route information; and

wherein the predetermined distance is adaptively determined based on at least one of a building boundary detected at the location of the user device, a road width detected at the location of the user device, a congestion level detected at the location of the user device, or a user speed.

2 . The user device of claim 1 , wherein the controller is configured to generate a synthesized image by synthesizing the POI represented as an AR-based 3D object to the surrounding image and to control the display module to display the synthesized image.

3 . The user device of claim 1 , wherein the database is configured to update the VPS map information based on the surrounding image.

4 . The user device of claim 3 , wherein the controller is configured to provide a weight to the surrounding image based on a feature of the surrounding image and to provide a reward point when the weight is higher than a predetermined reference.

5 . The user device of claim 4 , wherein the weight is provided based on a resolution, a capacity, or a length of the surrounding image, and the reward point varies depending on a sum of the weights.

6 . The user device of claim 1 , wherein the controller is configured to generate route information so that a time, a movement distance, or a congestion level is minimized.

7 . The user device of claim 6 , wherein the controller is configured to generate real- time route information based on the location of the user device and the delivery information.

8 . A control method of a user device, the control method comprising:

storing visual positioning system (VPS) map information and two dimensional (2D) map information, the VPS map information being based on augmented reality (AR);

receiving delivery information from a server;

photographing a surrounding image by an image sensor;

identifying a location of the user device based on the surrounding image and the VPS map information by using a VPS positioning model, the VPS positioning model being trained using metadata including the surrounding image and photographing direction information and excluding image data photographed in a direction of a ground;

generating 2D route information based on the delivery information and the 2D map information;

generating three dimensional (3D) route information based on the delivery information and the VPS map information; and

controlling a display module to display the 2D route information and the 3D route information based on an actual distance between the location of the user device and a location of a point of interest (POI) included in the delivery information;

wherein controlling the display module comprises:

based on the actual distance between the location of the user device and the location of the POI being greater than a predetermined distance, controlling the display module to display the 2D route information; and

based on the actual distance between the location of the user device and the location of the POI being less than or equal to the predetermined distance, controlling the display module to display the 3D route information; and

wherein the predetermined distance is adaptively determined based on at least one a building boundary detected at the location of the user device, a road width detected at the location of the user device, a congestion level detected at the location of the user device, a user speed.

9 . The control method of claim 8 , wherein controlling the display module comprises:

generating a synthesized image by synthesizing the POI represented as an AR-based 3D object to the surrounding image; and

controlling the display module to display the synthesized image.

10 . The control method of claim 8 , wherein storing the VPS map information and the 2D map information comprises updating the VPS map information based on the surrounding image.

11 . The control method of claim 10 , further comprising:

providing a weight to the surrounding image based on a feature of the surrounding image; and

providing a reward point when the weight is higher than a predetermined reference.

12 . The control method of claim 11 , wherein the weight is provided based on a resolution, a capacity, or a length of the surrounding image, and the reward point varies depending on a sum of the weights.

13 . The control method of claim 8 , wherein generating the 2D route information and the 3D route information comprises generating route information so that a time, a movement distance, or a congestion level is minimized.

14 . The control method of claim 13 , wherein generating the route information comprises generating real-time route information based on the location of the user device and the delivery information.

15 . A non-transitory computer-readable medium storing computer executable instructions when executed by a processor causing the processor to perform steps of the control method of the user device of claim 8 .

16 . A method comprising:

storing map information;

receiving destination information;

photographing a surrounding image by an image sensor;

identifying a current location of a user device based on the surrounding image and VPS map information that is based on augmented reality (AR) by using a VPS positioning model, the VPS positioning model being trained using metadata including the surrounding image and photographing direction information and excluding image data photographed in a direction of a ground;

generating two dimensional (2D) route information based on the stored map information and the destination information;

controlling a display module to display 2D route information;

while travelling toward the destination, adaptively determining a distance to reach the destination, the adaptively determined distance based on a road width detected at the current location, a congestion level detected at the current location, or a speed of the travelling;

generating three dimensional (3D) route information based on the destination information and visual positioning system (VPS) map information that is based on augmented reality (AR); and

in response to an actual distance to the destination being less than the adaptively determined distance, controlling the display module to display the 3D route information.

17 . The method of claim 16 , wherein storing the map information comprises storing the VPS map information and two dimensional (2D) map information;

wherein the 2D route information is generated based on the stored 2D map information; and

wherein the 3D route information is based on the stored VPS map information and the surrounding image obtained by the image sensor.

18 . The method of claim 16 , wherein the adaptively determined distance is based on a building boundary detected at the current location and the speed of the travelling.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (14)