Method for selecting surface points from a CAD model for locating industrial 3D objects, application of this method to the location of industrial 3D objects, and augmented reality system using 3d objects thus located

1 . A method for automatically selecting 3D points from a CAD model, for 3D locating of an industrial object, the method comprising the following steps:

generating a geometric model from the CAD model, comprising a sampling of a set of 3D points extracted from straight edges of the CAD model, so as to generate 3D surface points;

previously designating reference viewpoints from a priori knowledge of the current viewpoints of a camera;

acquiring, by way of a camera, a plurality of images taken from viewpoints of interest of the object;

associating each acquired image with a reference viewpoint in order to form partitions comprising sub-sets of images and identifying a reference image for each partition; and

selecting a number n>1 of 3D surface points for each partition, comprising the following steps:

refining the camera poses using the geometric model and identifying the associations of 3D surface points with 2D contour points that correspond to the refined poses;

calculating a score for each 3D surface point based on the number of times that 3D surface point is a match contour point;

projecting and clustering 3D surface points in each reference image; and

selecting the n 3D surface points having the best score for each of the partitions.

2 . The method of claim 1 , wherein the method is implemented in a convolutional neural network-based six-degree-of-freedom location, the method further comprising the steps of:

training a first convolutional neural network to recognize a viewpoint of interest of the object among a set of reference viewpoints; and

for each partition, learning heat maps corresponding to the 2D projections of the selected 3D surface points.

3 . The method of claim 2 , further comprising a step for identifying, by way of the first convolutional neural network, the closest viewpoint among six predefined viewpoints calculated automatically according to a 3D bounding box of the 3D object and camera calibration settings.

4 . The method of claim 3 , further comprising performing a partitioning sequence comprising the following steps:

annotating images for each learning sequence by launching 3D tracking;

associating the nearest reference viewpoint with each image based on 2D projections of 3D surface points and/or visibility tests; and

identifying a number p>1 of reference images acquired from the viewpoint closest to a same number p of reference viewpoints, thus producing a same number p of partitions of the acquired images.

5 . The method of claim 4 , further comprising a step of associating a partition with each image for the purpose of teaching a multi-class detector.

6 . The method of claim 2 , further comprising performing a learning sequence during which:

a multi-class detector is taught, allowing detection of the industrial object and prediction of its distribution; and

for each distribution, a keypoint detector is trained based on the location of heat maps.

7 . The method of claim 1 , further comprising generating a geometric model of an industrial 3D object offline, and then locating the industrial 3D object, locating the industrial 3D object comprising the following steps:

validating and refining a pose using the geometric model, this step being subject to prior initialization;

generating a 3D location of the industrial 3D object; and

constructing an appearance model of the industrial 3D object.

8 . The method of claim 7 , further comprising the following steps carried out online:

constructing an appearance model from the pose thus refined;

extracting features in the following frames and matching with the appearance model;

estimating a pose for the 3D object; and

determining a confidence score for the estimated pose.

9 . The method of claim 8 , wherein the pose estimation step is followed by a step that is conditioned by the level of the confidence score in the following way:

for a confidence score lower than a first predetermined threshold value, performing a 3D relocation based on a set of appearance models already produced;

for a confidence score between the first threshold value and a predetermined value, refining, then validating the pose using the geometric model;

for a confidence score greater than the second predetermined value, confirming the 3D location.

10 . The method of claim 9 , wherein the success or failure of the 3D relocation step leads respectively either to a new construction of an appearance model and a 3D location of the object, or to a reinitialization of the pose validation and refinement step.

11 . The method of claim 10 , wherein the initialization or reinitialization step prior to the execution of the pose validation and refinement step implements an initialization module based on a convolutional neural network architecture.

12 . An augmented reality system exploiting 3D objects located by implementing the surface point selection method according to claim 1 .