Method and apparatus with object estimation model training

1 . A processor-implemented method, comprising:

generating a cross-correlation loss based on a first feature vector, generated using an interim first neural network (NN) model provided an input based on first input data about a target object, and a second feature vector generated using a trained second neural network provided another input based on second input data about the target object; and

generating a trained first NN model, including training the interim first NN model based on the cross-correlation loss,

wherein the determining of the cross-correlation loss comprises:

determining a first cross-correlation loss based on a similarity of direction information between the first feature vector and the second feature vector;

determining a second cross-correlation loss based on a similarity of size information between the first feature vector and the second feature vector; and

determining the cross-correlation loss by weighted-summing the first cross-correlation loss and the second cross-correlation loss.

2 . The method of claim 1 , wherein a first data capture modality of the first input data is different from a second data capture modality of at least some of the second input data.

3 . The method of claim 1 ,

wherein the first input data comprises at least one image of the target object, and

wherein the second input data comprises at least one of point cloud data of the target object obtained from a lidar sensor or multi-modal data obtained from a plurality of modalities.

4 . The method of claim 1 ,

wherein the interim first NN model comprises a feature vector extractor and a detection head, and

wherein the training of the interim first NN model comprises training the feature vector extractor so that the cross-correlation loss is minimized.

5 . The method of claim 1 , wherein the training of the interim first NN model comprises performing unsupervised training of the interim first NN model without label information corresponding to the first input data.

6 . The method of claim 1 , further comprising:

obtaining a first additional feature vector by augmenting data of the first feature vector;

obtaining a second additional feature vector by augmenting data of the second feature vector;

obtaining a first target feature vector by concatenating the first feature vector with the first additional feature vector; and

obtaining a second target feature vector by concatenating the second feature vector with the second additional feature vector,

wherein the determining of the cross-correlation loss comprises determining the cross-correlation loss by comparing the first target feature vector with the second target feature vector.

7 . The method of claim 6 ,

wherein the obtaining of the first additional feature vector comprises obtaining the first additional feature vector by applying at least one of a dropout layer, in the interim first NN model, or random Gaussian noise to the first feature vector, and

wherein the obtaining of the second additional feature vector comprises obtaining the second additional feature vector through use of at least one of a dropout layer, of the second NN model, or through application of random Gaussian noise to the second feature vector.

8 . The method of claim 1 , further comprising:

obtaining first additional input data by augmenting data of the first input data;

obtaining a first additional feature vector by inputting the first additional input data to the interim first NN model;

obtaining second additional input data by augmenting data of the second input data;

obtaining a second additional feature vector by inputting the second additional input data to the second NN model;

obtaining a first target feature vector by concatenating the first feature vector with the first additional feature vector; and

obtaining a second target feature vector by concatenating the second feature vector with the second additional feature vector,

wherein the determining of the cross-correlation loss comprises determining the cross-correlation loss by comparing the first target feature vector with the second target feature vector.

9 . An electronic device comprising:

one or more processors configured to execute instructions; and

one or more memories storing the instructions,

wherein the execution of the instructions by the one or more processors configures the one or more processors to:

generate a cross-correlation loss based on a first feature vector, generated using an interim first neural network (NN) model provided an input based on first input data about a target object, and a second feature vector generated using a trained second NN model provided another input based on second input data about the target object; and

generate a trained first NN model, including training the interim first NN model based on the cross-correlation loss,

wherein, for the generating of the cross-correlation loss, the one or more processors are configured to:

determine a first cross-correlation loss based on a similarity of direction information between the first feature vector and the second feature vector;

determine a second cross-correlation loss based on a similarity of size information between the first feature vector and the second feature vector; and

determine the cross-correlation loss by weighted-summing the first cross-correlation loss and the second cross-correlation loss.

10 . The electronic device of claim 9 ,

wherein the first input data comprises at least one image of the target object captured by a camera, and

wherein the second input data comprises at least one of point cloud data of the target object obtained from a lidar sensor or multi-modal data obtained from a plurality of modalities.

11 . The electronic device of claim 10 , wherein the electronic device is vehicle, and the electronic device further comprises the camera and the lidar sensor.

12 . The electronic device of claim 9 ,

wherein the interim first NN model comprises a feature vector extractor and a detection head, and

wherein, for the training of the interim first NN model, the one or more processors are configured to train the feature vector extractor toward minimization of the cross-correlation loss.

13 . The electronic device of claim 9 , wherein, for the training of the interim first NN model, the one or more processors are configured to perform unsupervised training of the interim first NN model without label information corresponding to the first input data.

14 . The electronic device of claim 9 , wherein the one or more processors are further configured to:

obtain a first additional feature vector by augmenting data of the first feature vector;

obtain a second additional feature vector by augmenting data of the second feature vector;

obtain a first target feature vector by concatenating the first feature vector with the first additional feature vector; and

obtain a second target feature vector by concatenating the second feature vector with the second additional feature vector, and

wherein, for the generation of the cross-correlation loss, the one or more processors are configured to determine the cross-correlation loss by comparing the first target feature vector with the second target feature vector.

15 . The electronic device of claim 14 , wherein the one or more processors are further configured to:

obtain the first additional feature vector by applying at least one of a dropout layer, in the interim first NN model, or random Gaussian noise to the first feature vector; and

obtain the second additional feature vector by applying at least one of a dropout layer, of the second NN model, or through application of random Gaussian noise to the second feature vector.

16 . The electronic device of claim 9 , wherein the one or more processors are further configured to:

obtain first additional input data by augmenting data of the first input data;

obtain a first additional feature vector by inputting the first additional input data to the interim first NN model;

obtain second additional input data by augmenting data of the second input data;

obtain a second additional feature vector by inputting the second additional input data to the second NN model;

obtain a first target feature vector by concatenating the first feature vector with the first additional feature vector; and

obtain a second target feature vector by concatenating the second feature vector with the second additional feature vector,

wherein, further generation of the cross-correlation loss, the one or more processors are configured to determine the cross-correlation loss by comparing the first target feature vector with the second target feature vector.

17 . An electronic device comprising:

one or more sensors;

one or more processors configured to execute instructions; and

one or more memories storing the instructions,

wherein the execution of the instructions by the one or more processors configures the one or more processors to:

estimate the target object based on at least one image captured by an image sensor of the one or more sensors;

obtain a first feature vector by inputting the at least one image to a feature vector extractor of a first neural network (NN) model; and

estimate the target object by inputting the first feature vector to a detection head of the first NN model,

wherein the first NN model is a NN model having been trained based on a second NN model that was trained based on a modality different from the image sensor,

wherein the first NN model includes a camera-based three-dimensional (3D) object estimation detector, and

wherein the camera-based 3D object estimation detector is configured to estimate 3D location/posture and classification information of the target object using a feature extraction network model that estimates 3D geometric information of the target object around an adjacent object from the at least one image obtained from the one or more sensors.

18 . The electronic device of claim 17 ,

wherein the second NN model includes at least one of a lidar modal-based 3D object estimation detector or a multi-modal based 3D object estimation detector,

wherein the lidar modal-based 3D object estimation detector is configured to estimate 3D location information of the target object using point cloud data, and

wherein the multi-modal based 3D object estimation detector is configured to estimate the 3D location information of the target object using the one or more sensors.