Electronic device and method for obtaining three-dimensional (3D) skeleton data of object captured using plurality of cameras
An electronic device performs a method of obtaining Three-Dimensional (3D) skeleton data of an object obtained by using a first camera and a second camera. The method includes: obtaining a first image using the first camera and obtaining a second image using the second camera; obtaining, from the first image, a first Region Of Interest (ROI) comprising the object; obtaining, from the first ROI, first skeleton data comprising at least one keypoint of the object; obtaining a second ROI from the second image, based on the first skeleton data and information about a relative position between the first camera and the second camera; obtaining, from the second ROI, second skeleton data comprising at least one keypoint of the object; and obtaining 3D skeleton data of the object, based on the first skeleton data and the second skeleton data.
1 . A method, performed by an electronic device, of obtaining Three-Dimensional (3D) skeleton data of an object by using a first camera and a second camera, the method comprising:
obtaining a first image using the first camera and obtaining a second image using the second camera;
obtaining, from the first image, a first Region Of Interest (ROI) within the first image, the first ROI comprising the object;
obtaining, from the first ROI within the first image, first skeleton data comprising at least one keypoint of the object;
projecting 3D position coordinates of the at least one keypoint of the object to two-dimensional (2D) position coordinates on the second image, based on information about a relative position between the first camera and the second camera;
identifying, in the second image, second ROI having the 2D position coordinates of the at least one keypoint;
obtaining, from the second ROI, second skeleton data comprising the at least one keypoint of the object; and
obtaining the 3D skeleton data of the object, based on the first skeleton data and the second skeleton data.
2 . The method of claim 1 , wherein the object is a hand or a body part of a user.
3 . The method of claim 1 , wherein, in the obtaining of the first ROI from the first image, a first deep learning model trained to use the first image as an input value and output an image region comprising the object as the first ROI is used.
4 . The method of claim 1 , wherein, in the obtaining of the first skeleton data from the first ROI, a second deep learning model trained to use the first ROI as an input value and output, as the first skeleton data, a graph comprising the at least one keypoint of the object as a joint is used.
5 . The method of claim 1 , wherein the first skeleton data comprises 3D position coordinates of the at least one keypoint with respect to a preset origin in space.
6 . The method of claim 1 , wherein the second skeleton data comprises 2D position coordinates of the at least one keypoint with respect to a preset origin on a plane of the second image.
7 . The method of claim 6 , wherein, in the obtaining of the second skeleton data from the second ROI, a third deep learning model trained to use the second ROI as an input value and output, as the second skeleton data, a graph comprising the at least one keypoint of the object as a joint is used.
8 . The method of claim 6 , wherein the first skeleton data comprises 3D position coordinates of the at least one keypoint with respect to a preset origin in space, and
wherein the obtaining of the 3D skeleton data of the object based on the first skeleton data and the second skeleton data comprises:
projecting the 3D position coordinates of the at least one keypoint, the 3D position coordinates being included in the first skeleton data, to 2D position coordinates on a plane of the first image; and
obtaining 3D position coordinates, based on the 2D position coordinates projected on the plane of the first image and the 2D position coordinates included in the second skeleton data.
9 . The method of claim 1 , wherein the second skeleton data comprises 3D position coordinates of the at least one keypoint with respect to a preset origin in space.
10 . The method of claim 9 , wherein, in the obtaining of the second skeleton data from the second ROI, a second deep learning model trained to use the second ROI as an input value and output, as the second skeleton data, a graph comprising the at least one keypoint of the object as a joint is used.
11 . The method of claim 9 , wherein the first skeleton data comprises 3D position coordinates of the at least one keypoint with respect to a preset origin in space, and
wherein the obtaining of the 3D skeleton data of the object based on the first skeleton data and the second skeleton data comprises obtaining 3D position coordinates of the at least one keypoint, based on an average value of the 3D position coordinates included in the first skeleton data and the 3D position coordinates included in the second skeleton data.
12 . The method of claim 1 , wherein the obtaining of the 3D skeleton data of the object based on the first skeleton data and the second skeleton data comprises obtaining 3D position coordinates of the at least one keypoint, based on a value obtained by weight-combining position coordinates in the first skeleton data with position coordinates in the second skeleton data.
13 . An electronic device comprising:
a first camera;
a second camera;
a storage storing at least one instruction; and
at least one processor configured to electrically connect with the first camera and the second camera and configured to execute the at least one instruction stored in the storage to:
obtain a first image via the first camera and obtain a second image via the second camera,
obtain, from the first image, a first Region Of Interest (ROI) within the first image, the first ROI comprising an object,
obtain, from the first ROI within the first image, first skeleton data comprising at least one keypoint of the object,
projecting three-dimensional (3D) position coordinates of the at least one keypoint of the object to two-dimensional (2D) position coordinates on the second image, based on information about a relative position between the first camera and the second camera,
identifying, in the second image, second ROI having the 2D position coordinates of the at least one keypoint,
obtain, from the second ROI, second skeleton data comprising the at least one keypoint of the object, and
obtain the 3D skeleton data of the object, based on the first skeleton data and the second skeleton data.
14 . A non-transitory computer-readable recording medium having recorded thereon a program for performing the method of claim 1 , on a computer.