Data processing apparatus and method for determining a pose

1 . A data processing apparatus for determining a pose based on an image of a three dimensional (3D) scene, wherein the data processing apparatus comprises:

a processor; and

a memory storing instructions that when executed by the processor cause the data processing apparatus to perform steps including:

select a plurality of key two dimensional (2D) points of a plurality of 2D points of the image based on a respective score of each of the plurality of 2D points, wherein the score is one of a repeatability score and a reliability score;

determine at least for a subset of the plurality of 2D points of the image a respective feature vector for obtaining a plurality of feature vectors;

concatenate the image with the plurality of feature vectors for obtaining an intermediate tensor;

determine a plurality of 3D points of the 3D scene based on the intermediate tensor;

determine the pose based on the plurality of key 2D points of the image and the plurality of 3D points of the 3D scene using a Perspective-n-Point scheme;

implement a further neural network to determine a score for selecting the plurality of key 2D points of the plurality of 2D points of the image based on the respective score of each of the plurality of 2D points; and

train the further neural network using a target heat map based on a training image, wherein a plurality of maximum values of the target heat map correspond to a plurality of locations of projections of a plurality of 3D points of a 3D model of the 3D scene.

2 . The data processing apparatus of claim 1 , wherein the processing circuitry is configured to determine the pose based on the plurality of key 2D points of the image and the plurality of 3D points of the 3D scene using the Perspective-n-Point scheme and a random sample consensus (RANSAC) scheme.

3 . The data processing apparatus of claim 1 , wherein the image comprises one or more image channels.

4 . The data processing apparatus of claim 1 , wherein the respective score determined by the processing circuitry for each of the plurality of 2D points is a confidence score.

5 . The data processing apparatus of claim 1 , wherein the processing circuitry is configured to implement a neural network configured to determine the plurality of 3D points of the 3D scene based on the intermediate tensor.

6 . The data processing apparatus of claim 1 , wherein the further neural network is further configured to determine at least for the subset of the plurality of 2D points of the image the respective feature vector.

7 . The data processing apparatus of claim 1 , wherein the processing circuitry is configured to implement a neural network configured to determine the plurality of 3D points of the 3D scene based on the intermediate tensor and to determine a score for selecting the plurality of key 2D points of the plurality of 2D points of the image based on the respective score of each of the plurality of 2D points.

8 . The data processing apparatus of claim 7 , wherein the processing circuitry is configured to train the neural network using a target heat map based on a training image, wherein a plurality of maximum values of the target heat map correspond to a plurality of locations of projections of a plurality of 3D points of a 3D model of the 3D scene.

9 . The data processing apparatus of claim 1 , wherein the processing circuitry is configured to concatenate the image with the plurality of feature vectors for obtaining the intermediate tensor by concatenating for each 2D point an intensity value of the 2D point with a feature vector of the 2D point.

10 . A data processing method for determining a pose based on an image of a three dimensional (3D) scene, wherein the data processing method comprises:

selecting a plurality of key two dimensional (2D) points of a plurality of 2D points of the image based on a respective score of each of the plurality of 2D points, wherein the score is one of a repeatability score and a reliability score;

determining at least for a subset of the plurality of 2D points of the image a respective feature vector for obtaining a plurality of feature vectors;

concatenating the image with the plurality of feature vectors for obtaining an intermediate tensor;

determining a plurality of 3D points of the 3D scene based on the intermediate tensor; and

determining the pose based on the plurality of key 2D points of the image and the plurality of 3D points of the 3D scene using a Perspective-n-Point scheme;

implementing a further neural network to determine a score for selecting the plurality of key 2D points of the plurality of 2D points of the image based on the respective score of each of the plurality of 2D points; and

training the further neural network using a target heat map based on a training image, wherein a plurality of maximum values of the target heat map correspond to a plurality of locations of projections of a plurality of 3D points of a 3D model of the 3D scene.

11 . The data processing method of claim 10 , further comprising:

determining the pose based on the plurality of key 2D points of the image and the plurality of 3D points of the 3D scene using the Perspective-n-Point scheme and a random sample consensus (RANSAC) scheme.

12 . The data processing method of claim 10 , wherein the image comprises one or more image channels.

13 . The data processing method of claim 10 , wherein the respective score for each of the plurality of 2D points is a confidence score.

14 . The data processing method of claim 10 , further comprising:

implementing a neural network configured to determine the plurality of 3D points of the 3D scene based on the intermediate tensor.

15 . The data processing method of claim 10 , wherein the further neural network is further configured to determine at least for the subset of the plurality of 2D points of the image the respective feature vector.

16 . A non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are run on a processor, the processor is caused to:

selecting a plurality of key two dimensional (2D) points of a plurality of 2D points of the image based on a respective score of each of the plurality of 2D points, wherein the score is one of a repeatability score and a reliability score;

determining at least for a subset of the plurality of 2D points of the image a respective feature vector for obtaining a plurality of feature vectors;

concatenating the image with the plurality of feature vectors for obtaining an intermediate tensor;

determining a plurality of 3D points of the 3D scene based on the intermediate tensor; and

determining the pose based on the plurality of key 2D points of the image and the plurality of 3D points of the 3D scene using a Perspective-n-Point scheme;

implementing a further neural network to determine a score for selecting the plurality of key 2D points of the plurality of 2D points of the image based on the respective score of each of the plurality of 2D points; and

training the further neural network using a target heat map based on a training image, wherein a plurality of maximum values of the target heat map correspond to a plurality of locations of projections of a plurality of 3D points of a 3D model of the 3D scene.