Apparatus and method for image-based positioning to predict a camera pose from image data

1. A method for image-based positioning to predict a camera pose from image data, comprising:

obtaining, by a positioning apparatus, a prediction result indicating which cell the image data belongs to among cells constituting a space for positioning, wherein the prediction result includes at least one among a first prediction result obtained through a hard classification-based learning classification network and a second prediction result obtained through a soft classification-based learning classification network;

obtaining, by the positioning apparatus, map information on the space for positioning by using a cell index selected based on the prediction result, wherein the map information includes minimum and maximum values of coordinates constituting a cell and a cell size for each cell index and the cell index is in a form of coordinates; and

calculating, by the positioning apparatus, a position of the image data based on the map information and outputting a corresponding camera pose,

wherein the position of the image data is based on at least one among a first position and a second position, and the first position is calculated according to map information obtained based on the first prediction result and the second position is calculated according to map information obtained based on the second prediction result.

2. The method of claim 1 , wherein

the outputting of a camera pose comprises

the first position or the second position is calculated based on a sum of weights of neighboring cells of the selected cell index in the space for positioning.

3. The method of claim 1 , wherein

a cell constituting the space for positioning is mapped to one class, and

the prediction result includes confidence for each class,

wherein the outputting of a camera pose comprises

outputting of the camera pose and confidence corresponding thereto.

4. The method of claim 3 , wherein

the obtaining of map information comprises:

selecting a class having highest confidence among the confidence for each class, and obtaining a cell index mapped to the selected class; and

obtaining map information on the space for the positioning by using the obtained cell index.

5. The method of claim 1 , wherein

the hard classification-based learning classification network is learned through hard classification-based learning,

the hard classification-based learning is performed by converting training data into an index for applying camera pose classification and performing hard labeling for learning on each index, and the hard labeling is performed by setting only one index cell corresponding to a camera pose to “1” and setting the rest to “0”.

6. The method of claim 1 , wherein

a cell constituting the space for positioning is mapped to one class,

the prediction result includes a score for each class, and the score is an evaluation score calculated based on a loss function,

wherein the obtaining of map information comprises:

selecting a class having a highest score among the scores for each class, and obtaining a cell index mapped to the selected class; and

obtaining map information on the space for the positioning by using the obtained cell index.

7. The method of claim 1 , wherein

the soft classification-based learning classification network is learned through soft classification-based learning,

the soft classification-based learning is performed by converting training data into indexes for applying camera pose classification and performing soft labeling for learning on each index, and the soft labeling is based on a linear interpolation method.

8. The method of claim 7 , wherein

the soft labeling determines a soft label based on a distance and an area of a neighboring cell adjacent to an absolute location.

9. An apparatus for image-based positioning to predict a camera pose from image data, comprising:

an interface device configured to receive image data for image-based positioning;

a memory in which map information on cells constituting a space for positioning is stored, wherein the map information includes minimum and maximum values of coordinates constituting a cell and a cell size for each cell index and the cell index is in a form of coordinates; and

a processor for predicting a camera pose based on the image data, wherein

the processor performs operations by:

obtaining, through the interface device, a prediction result indicating which cell the image data belongs to among cells constituting a space for positioning, wherein the prediction result includes at least one among a first prediction result obtained through a hard classification-based learning classification network and a second prediction result obtained through a soft classification-based learning classification network;

obtaining map information on the space for positioning by using a cell index selected based on the prediction result; and

calculating a location for the image data based on the map information and outputting a corresponding camera pose,

wherein the position of the image data is based on at least one among a first position and a second position, and the first position is calculated according to map information obtained based on the first prediction result and the second position is calculated according to map information obtained based on the second prediction result.

10. The apparatus of claim 9 , wherein

the processor calculates the first position or the second position based on a sum of weights of neighboring cells of the selected cell index in the space for positioning when performing the operation of outputting a camera pose.

11. The apparatus of claim 9 , wherein cells constituting the space for positioning are mapped to one class, and the prediction result includes confidence for each class, wherein the processor outputs the camera pose and confidence corresponding thereto when performing an operation of outputting a camera pose.

12. The apparatus of claim 11 , wherein

when performing the operation of obtaining map information,

the processor performs the operation by:

selecting a class having highest confidence among the confidence for each class, and obtaining a cell index mapped to the selected class; and

obtaining map information on the space for positioning by using the obtained cell index.

13. The apparatus of claim 9 , wherein

the hard classification-based learning classification network is learned through hard classification-based learning,

the hard classification-based learning is performed by converting training data into an index for applying camera pose classification and performing hard labeling for learning on each index, and the hard labeling is performed by setting only one index cell corresponding to a camera pose to “1” and setting the rest to “0”.

14. The apparatus of claim 9 , wherein

a cell constituting the space for positioning is mapped to one class,

the prediction result includes a score for each class, and the score is an evaluation score calculated based on a loss function,

wherein when performing the operation of obtaining map information,

the processor performs the operation by:

selecting a class having a highest score among the scores for each class, and obtaining a cell index mapped to the selected class; and

obtaining map information on the space for the positioning by using the obtained cell index.

15. The apparatus of claim 9 , wherein

the soft classification-based learning classification network is learned through soft classification-based learning,

the soft classification-based learning is performed by converting training data into indexes for applying camera pose classification and performing soft labeling for learning on each index, and the soft labeling is based on a linear interpolation method.

16. The apparatus of claim 9 , wherein

the processor comprises:

a hard classification layer unit for outputting a hard classification result including a camera pose predicted for the image data and confidence corresponding thereto by performing the operation of obtaining a prediction result by using the hard classification-based learning classification network, the operation of obtaining map information, and the operation of outputting a corresponding camera pose;

a soft classification layer unit for outputting a soft classification result including a camera pose predicted for the image data by performing the operation of obtaining a prediction result by using the soft classification-based learning classification network, the operation of obtaining map information, and the operation of outputting a corresponding camera pose; and

a fusion processing unit for outputting a final camera pose predicted for the image data by converging the hard classification result and the soft classification result.

17. The apparatus of claim 9 , wherein

the processor comprises:

a hard classification layer unit for outputting a hard classification result including a camera pose predicted for the image data and confidence corresponding thereto by performing the operation of obtaining a prediction result by using the hard classification-based learning classification network, the operation of obtaining map information, and the operation of outputting a corresponding camera pose;

a regression layer unit for outputting a regression estimation result including a camera pose predicted for the image data through regression estimation; and

a fusion processing unit for outputting a final camera pose predicted for the image data by converging the hard classification result and the regression estimation result.

18. The apparatus of claim 9 , wherein

the processor comprises:

a soft classification layer unit for outputting a soft classification result including a camera pose predicted for the image data by performing the operation of obtaining a prediction result by using the soft classification-based learning classification network, the operation of obtaining map information, and the operation of outputting a corresponding camera pose;

a regression layer unit for outputting a regression estimation result including a camera pose predicted for the image data through regression estimation; and

a fusion processing unit for outputting a final camera pose predicted for the image data by converging the soft classification result and the regression estimation result.

19. The apparatus of claim 9 , wherein

the processor comprises:

a hard classification layer unit for outputting a hard classification result including a camera pose predicted for the image data and confidence corresponding thereto by performing the operation of obtaining a prediction result by using the hard classification-based learning classification network, the operation of obtaining map information, and the operation of outputting a corresponding camera pose;

a soft classification layer unit for outputting a soft classification result including a camera pose predicted for the image data by performing the operation of obtaining a prediction result by using the soft classification-based learning classification network, the operation of obtaining map information, and the operation of outputting a corresponding camera pose;

a regression layer unit for outputting a regression estimation result including a camera pose predicted for the image data through regression estimation; and

a fusion processing unit for outputting a final camera pose predicted for the image data by converging the soft classification result, the hard classification result, and the regression estimation result.