Localization method and apparatus based on 3D color map

1. A localization method comprising:

generating a synthetic image corresponding to a current time period based on a three-dimensional (3D) color map corresponding to an image captured at a time period prior to the current time period, and position information at the current time period by projecting the 3D color map onto a 3D semantic map; and

determining final position information at the current time period based on the synthetic image corresponding to the current time period and an image captured at the current time period.

2. The localization method of claim 1 , wherein the generating of the synthetic image comprises generating the synthetic image by projecting the 3D color map corresponding to the image at the prior time period onto the 3D semantic map based on the position information at the current time period.

3. The localization method of claim 1 , wherein the determining of the final position comprises:

calculating relative pose information based on the synthetic image corresponding to the current time period and the image captured at the current time period; and

determining the final position information at the current time period based on the relative pose information.

4. The localization method of claim 3 , wherein the calculating of the relative pose position comprises:

matching the synthetic image corresponding to the current time period and the image captured at the current time period;

comparing first pose information in the synthetic image corresponding to the current time period to second pose information in the image captured at the current time period; and

calculating the relative pose information based on a comparison result.

5. The localization method of claim 4 , wherein the comparing of the first pose information comprises at least one of:

comparing a color of the synthetic image corresponding to the current time period to a color of the image captured at the current time period;

comparing semantic information of the synthetic image corresponding to the current time period to semantic information of the image captured at the current time period; and

comparing depth information of the synthetic image corresponding to the current time period to depth information of the image captured at the current time period.

6. The localization method of claim 3 , wherein the determining of the final position information comprises determining the final position information at the current time period by modifying the position information at the current time period based on the relative pose information.

7. The localization method of claim 1 , further comprising:

generating a 3D color map corresponding to the image captured at the current time period based on the image at the current time period, the final position information at the current time period, and a 3D geometry acquired from the 3D semantic map.

8. The localization method of claim 7 , wherein the generating of the 3D color map comprises generating the 3D color map corresponding to the image captured at the current time period such that a color difference between the image captured at the current time period and the synthetic image corresponding to the current time period is within a desired standard based on the final position information at the current time period.

9. The localization method of claim 7 , wherein the generating of the 3D color map comprises:

generating a projected image by projecting the 3D semantic map onto a projection surface corresponding to the final position information at the current time period;

coloring the projected image based on the image captured at the current time period; and

generating the 3D color map corresponding to the image captured at the current time period by inversely projecting the colored projected image onto the 3D semantic map.

10. The localization method of claim 9 , wherein the generating of the projected image comprises:

extracting a partial region of the 3D semantic map based on a parameter of a camera that captures the image at the current time period; and

projecting the partial region onto the projection surface.

11. The localization method of claim 9 , wherein the coloring of the projected image comprises:

determining a color of pixels included in the projected image based on a mapping relationship between the projected image and the image captured at the current time period.

12. The localization method of claim 11 , wherein the pixels correspond to surface points corresponding to a surface of a road.

13. The localization method of claim 7 , further comprising:

rendering a virtual object at the current time period with the 3D color map corresponding to the image captured at the current time period.

14. The localization method of claim 13 , wherein the rendering of the virtual object comprises:

acquiring a background color displayed on a transparent display based on the 3D color map corresponding to the image captured at the current time period; and

rendering the virtual object based on a color of the virtual object and the background color.

15. The localization method of claim 13 , wherein the rendering of the virtual object comprises rendering a shadow component of the virtual object and an inter-reflected component by the virtual object with the 3D color map corresponding to the image captured at the current time period.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the localization method of claim 1 .

17. A localization apparatus comprising:

one or more sensors configured to capture an image at a first time period and position information at a second time period; and

a processor configured to:

generate a synthetic image corresponding to the second time period based on a three-dimensional (3D) color map corresponding to the image captured at the first time period and the position information at the second time period by projecting the 3D color mat onto a 3D semantic map, and

determine final position information at the second time period based on the synthetic image corresponding to the second time period and the image at the second time period.

18. The localization apparatus of claim 17 , wherein the processor is configured to generate the synthetic image by projecting the 3D color map corresponding to the image captured at the first time period onto the 3D semantic map based on the position information at the second time period.

19. The localization apparatus of claim 17 , wherein the processor is configured to calculate relative pose information based on the synthetic image corresponding to the second time period and the image captured at the second time period, and to determine the final position information at the second time period based on the relative pose information.

20. The localization apparatus of claim 17 , wherein the processor is configured to generate a 3D color map corresponding to the image captured at the second time period based on the image captured at the second time period, the final position information at the second time period, and a 3D geometry acquired from the 3D semantic map.

21. The localization apparatus of claim 20 , wherein the processor is configured to generate a projected image by projecting the 3D semantic map onto a projection surface corresponding to the final position information at the second time period, to color the projected image based on the image at the second time period, and to generate the 3D color map corresponding to the image at the second time period by inversely projecting the colored projected image onto the 3D semantic map.

22. A localization method comprising:

receiving a current image frame and position information related to the current image frame;

generating a synthetic image corresponding to the current image frame based on the position information and a three-dimensional (3D) color map corresponding to a previous image frame by projecting the 3D color map onto a 3D semantic map; and

determining a final position of a current image of the current image frame based on a pose difference between the synthetic image and the current image.

23. The method of claim 22 , wherein the pose difference corresponds to a difference between first pose information in the synthetic image corresponding to the current image frame, and second pose information in the current image frame.