Method and electronic device for training neural network model by augmenting image representing object captured by multiple cameras
Provided is a computer-implemented method of training a neural network model by augmenting images representing objects. The method includes: obtaining a first object recognition result predicted by a first neural network model using, as an input, a first image captured by a first camera capturing, from a first viewpoint, a space including at least one object; converting the obtained first object recognition result, based on a conversion relationship between a first camera coordinate system corresponding to the first camera and a second camera coordinate system corresponding to a second camera capturing, from a second viewpoint, the space; generating, based on the first object recognition result converted with respect to the second viewpoint, training data by performing labeling on a second image that corresponds to the first image, the second image being captured by the second camera; and training a second neural network model by using the generated training data.
1 . A computer-implemented method of training a neural network model by augmenting images representing objects captured by multiple cameras, the computer-implemented method comprising:
obtaining a first object recognition result predicted by a first neural network model using, as an input, a first image captured by a first camera facing, from a first viewpoint, a space including at least one object;
capturing a second image by a second camera facing the space from a second viewpoint;
obtaining a label for the second image by converting the obtained first object recognition result, based on a conversion relationship between a first camera coordinate system corresponding to the first camera and a second camera coordinate system corresponding to the second camera;
generating, based on the first object recognition result converted with respect to the second viewpoint, training data by performing labeling, using the obtained label for the second image, on the second image that corresponds to the first image; and
training a second neural network model by using the generated training data.
2 . The computer-implemented method of claim 1 , wherein the converting of the obtained first object recognition result comprises, based on the conversion relationship, converting three-dimensional pose information into two-dimensional pose information,
wherein the three-dimensional pose information comprises spatial coordinates of feature points representing a position and a direction of an object identified in the first image,
wherein the two-dimensional pose information comprises image coordinates of feature points representing a position of the object identified in the second image, and
wherein the generating of the training data comprises generating the training data by performing labeling on the second image such that the two-dimensional pose information included in the first object recognition result converted with respect to the second viewpoint is used as two-dimensional pose information of the object in the second image.
3 . The computer-implemented method of claim 2 , wherein the converting of the obtained first object recognition result further comprises:
based on a conversion relationship between the first camera coordinate system and a world coordinate system, converting the three-dimensional pose information in the first camera coordinate system into three-dimensional pose information in the world coordinate system;
based on a conversion relationship between the world coordinate system and the second camera coordinate system, converting the three-dimensional pose information in the world coordinate system into three-dimensional pose information in the second camera coordinate system; and
converting the three-dimensional pose information in the second camera coordinate system into two-dimensional pose information in a second image coordinate system of the second camera by projecting spatial coordinates in the second camera coordinate system to the second image coordinate system of the second camera.
4 . The computer-implemented method of claim 3 , wherein the conversion relationship between the first camera coordinate system and the world coordinate system is obtained by performing first camera calibration to obtain internal parameters of the first camera and external parameters of the first camera using matching pairs of spatial coordinates of a number of feature points in the world coordinate system and image coordinates of the feature points in a first image coordinate system of the first camera, and
wherein the feature points are detected by moving an electronic device within a field of view of the first camera and detecting a center point of a detection area, a marker attached to the electronic device, or an appearance of the electronic device.
5 . The computer-implemented method of claim 2 , wherein the generating of the training data further comprises generating the training data by performing labeling on the second image such that, among the two-dimensional pose information included in the first object recognition result converted with respect to the second viewpoint, a first part corresponding to the second image is distinguished from a second part that does not correspond to the second image.
6 . The computer-implemented method of claim 1 , wherein, based on a plurality of second images that are captured by the second camera from the second viewpoint, the generating of the training data comprises generating the training data by performing labeling on at least one of the plurality of second images, based on the first object recognition result converted with respect to the second viewpoint.
7 . The computer-implemented method of claim 6 , wherein the plurality of second images captured from the second viewpoint are captured by capturing a dynamic object at preset time intervals, and
wherein the second camera is configured to maintain the second viewpoint.
8 . The computer-implemented method of claim 1 , wherein, based on a plurality of second images that are captured by the second camera respectively from a plurality of different second viewpoints, the converting comprises converting the obtained first object recognition result, based on a conversion relationship between the first camera coordinate system and the second camera coordinate system for at least one of the plurality of second viewpoints, and
wherein the generating of the training data comprises generating the training data by performing labeling on at least one of the plurality of second images, based on the first object recognition result converted with respect to the at least one of the plurality of second viewpoints.
9 . The computer-implemented method of claim 8 , wherein the plurality of second images captured respectively from the plurality of different second viewpoints are captured from the plurality of different second viewpoints, and
wherein the second camera mounted on an electronic device is configured to circle around a static object, or changes a distance to the static object.
10 . The computer-implemented method of claim 1 , wherein the generating of the training data comprises generating the training data, based on an electronic device equipped with the second camera, which is configured to recognize an object at a lower level than a preset standard, or to move to a position specified in a spatial map used by the electronic device.
11 . An electronic device communicated with an external device including a first camera, comprising:
a second camera;
a communication unit;
a memory configured to store at least one instruction;
at least one processor operatively connected to the second camera, the communication unit, and the memory and configured to execute the at least one instruction to;
obtain, through the communication unit, a first object recognition result predicted by a first neural network model using, as an input, a first image captured from a first viewpoint by the first camera of the external device facing a space including at least one object,
capture a second image by the second camera facing the space from a second viewpoint,
obtain a label for the second image by converting the obtained first object recognition result, based on a conversion relationship between a first camera coordinate system corresponding to the first camera of the external device and a second camera coordinate system corresponding to the second camera,
generate training data by performing labeling, using the obtained label for the second image, on the second image that corresponds to the first image, based on the first object recognition result converted with respect to the second viewpoint, and
train a second neural network model by using the generated training data.
12 . The electronic device of claim 11 , wherein the at least one processor is further configured to execute the at least one instruction to:
based on the conversion relationship, convert three-dimensional pose information into two-dimensional pose information,
wherein the three-dimensional pose information comprises spatial coordinates of feature points representing a position and a direction of an object identified in the first image, and
wherein the two-dimensional pose information comprises image coordinates of feature points representing a position of the object identified in the second image, and
generate the training data by performing labeling on the second image such that the two-dimensional pose information included in the first object recognition result converted with respect to the second viewpoint is used as two-dimensional pose information of the object in the second image.
13 . The electronic device of claim 12 , wherein the at least one processor is further configured to execute the at least one instruction to:
based on a conversion relationship between the first camera coordinate system and a world coordinate system, convert the three-dimensional pose information in the first camera coordinate system into three-dimensional pose information in the world coordinate system,
based on a conversion relationship between the world coordinate system and the second camera coordinate system, convert the three-dimensional pose information in the world coordinate system into three-dimensional pose information in the second camera coordinate system, and
convert the three-dimensional pose information in the second camera coordinate system into two-dimensional pose information in a second image coordinate system of the second camera by projecting spatial coordinates in the second camera coordinate system to the second image coordinate system of the second camera.
14 . The electronic device of claim 13 , wherein the at least one processor is further configured to execute the at least one instruction to obtain the conversion relationship between the first camera coordinate system and the world coordinate system by performing first camera calibration to obtain internal parameters of the second camera of the external device and external parameters of the second camera of the external device using matching pairs of spatial coordinates of a number of feature points in the world coordinate system and image coordinates of the feature points in the second image coordinate system of the second camera of the external device, and
wherein the feature points are detected by moving the electronic device within a field of view of the second camera of the external device and detecting a center point of a detection area, a marker attached to the electronic device, or an appearance of the electronic device.
15 . The electronic device of claim 12 , wherein the at least one processor is further configured to execute the at least one instruction to generate the training data by performing labeling on the second image such that, among the two-dimensional pose information included in the first object recognition result converted with respect to the second viewpoint, a first part corresponding to the second image is distinguished from a second part that does not correspond to the second image.
16 . The electronic device of claim 11 , wherein the at least one processor is further configured to execute the at least one instruction to, based on a plurality of second images that are captured by the first camera from the second viewpoint, generate the training data by performing labeling on at least one of the plurality of second images, based on the first object recognition result converted with respect to the second viewpoint.
17 . The electronic device of claim 11 , wherein the at least one processor is further configured to execute the at least one instruction to:
based on a plurality of second images that are captured by the first camera respectively from a plurality of different second viewpoints, convert the obtained first object recognition result, based on a conversion relationship between the first camera coordinate system and the second camera coordinate system for each of the plurality of second viewpoints, and
generate the training data by performing labeling on at least one of the plurality of second images, based on the first object recognition result converted with respect to each of the plurality of second viewpoints.
18 . The electronic device of claim 11 , wherein the at least one processor is further configured to execute the at least one instruction to generate the training data, and
wherein the electronic device is configured to recognize an object at a lower level than a preset standard, or to move to a position specified in a spatial map used by the electronic device.
19 . A cloud server comprising:
a communication unit;
a memory storing at least one instruction;
at least one processor operatively connected to the communication unit and the memory and configured to execute the at least one instruction to:
obtain a first object recognition result predicted by a first neural network model using, as an input, a first image captured by a first camera facing a space including at least one object from a first viewpoint,
obtain, through the communication unit, a second image captured by a second camera facing the space from a second viewpoint,
obtain a label for the second image by converting the obtained first object recognition result, based on a conversion relationship between a first camera coordinate system corresponding to the first camera and a second camera coordinate system corresponding to the second camera,
generate training data by performing labeling, using the obtained label for the second image, on the second image corresponding to the first image, based on the first object recognition result converted with respect to the second viewpoint, and
train a second neural network model by using the generated training data.
20 . The cloud server of claim 19 , wherein the at least one processor is further configured to execute the at least one instruction to transmit, through the communication unit, network parameter values of the trained second neural network model to an electronic device with the second neural network model.