Three-dimensional model processing method and three-dimensional model processing apparatus
A three-dimensional model processing method includes: generating, from first images shot by respective cameras at a first time, a first three-dimensional model including: first three-dimensional points indicating a subject at the first time; and first camera parameters indicating positions and orientations of the cameras; generating, from second images shot by the respective cameras at a second time, a second three-dimensional model including: second three-dimensional points indicating the subject at the second time; and second camera parameters indicating positions and orientations of the cameras; detecting a stationary camera among the cameras, whose position and orientation has not changed between the first and second times, or stationary three-dimensional points among the three-dimensional points, whose positions have not changed between the first and second times; and matching world coordinate systems of the first and second three-dimensional models, based on the first camera parameters of the stationary camera or the stationary three-dimensional points detected.
1. A processing method for a three-dimensional model, comprising:
generating a first three-dimensional model from first images shot by respective cameras at a first time, the first three-dimensional model including: first three-dimensional points indicating a subject at the first time; and first camera parameters indicating positions and orientations of the cameras;
generating a second three-dimensional model from second images shot by the respective cameras at a second time different from the first time, the second three-dimensional model including: second three-dimensional points indicating the subject at the second time; and second camera parameters indicating positions and orientations of the cameras;
detecting (1) one or more stationary cameras among the cameras, positions and orientations of the one or more stationary cameras not having changed between the first time and the second time, or (2) one or more stationary three-dimensional points among the plurality of three-dimensional points, positions of the one or more stationary three-dimensional points not having changed between the first time and the second time; and
matching a first world coordinate system of the first three-dimensional model and a second world coordinate system of the second three-dimensional model, based on the first camera parameters of the one or more stationary cameras or the one or more stationary three-dimensional points detected.
2. The processing method according to claim 1 , wherein
the first world coordinate system and the second world coordinate system are matched based on both the first camera parameters of the one or more stationary cameras and the one or more stationary three-dimensional points detected.
3. The processing method according to claim 1 , wherein
in the detecting:
a global vector for a pair of images consisting of an image included in the first images and an image included in the second images is calculated, the pair of images being shot by a same camera among the cameras; and
whether the position and the orientation of the camera that shot the pair of images have changed between the first time and the second time is detected using the global vector calculated.
4. The processing method according to claim 1 , wherein
in the detecting:
a motion vector of the subject in a pair of images consisting of an image included in the first images and an image included in the second images is calculated, the pair of images being shot by a same camera among the cameras; and
whether a position of a three-dimensional point corresponding to the subject has changed between the first time and the second time is determined using the motion vector detected.
5. The processing method according to claim 1 , wherein
in the detecting, whether the position and the orientation of each of the cameras has changed is determined based on a result of sensing by an acceleration sensor included in each of the cameras.
6. The processing method according to claim 1 , wherein
in the matching:
two points are selected from the one or more stationary three-dimensional points and three-dimensional positions of the one or more stationary cameras; and
a deviation in scale between the first three-dimensional model and the second three-dimensional model is corrected based on a distance between the two points selected, in each of the first three-dimensional model and the second three-dimensional model.
7. The processing method according to claim 1 , wherein
in the matching:
one stationary camera is selected from the one or more stationary cameras; and
a deviation in rotation or translation between the first three-dimensional model and the second three-dimensional model is corrected based on a three-dimensional position and an orientation of the one stationary camera selected, in each of the first three-dimensional model and the second three-dimensional model.
8. The processing method according to claim 1 , wherein
in the matching, at least one of a rotation, a translation, or a scale of the first three-dimensional model or the second three-dimensional model is corrected to maintain relative relationships between the plurality of three-dimensional points and the positions of the cameras in the first three-dimensional model and relative relationships between the plurality of three-dimensional points and the positions of the cameras in the second three-dimensional model, and to reduce error between positions of the one or more stationary three-dimensional points and positions of the one or more stationary cameras in the first three-dimensional model and the second three-dimensional model.
9. The processing method according to claim 1 , wherein
in the matching, a coordinate matching scheme is selected based on a third reliability which is based on at least one of (i) the first camera parameters of the one or more stationary cameras and a first reliability of the one or more three-dimensional points included in the first three-dimensional model or (ii) the second camera parameters of the one or more stationary cameras and a second reliability of the three-dimensional points included in the second three-dimensional model, and the first world coordinate system of the first three-dimensional model and the second world coordinate system of the second three-dimensional model are matched by using the coordinate matching scheme selected.
10. The processing method according to claim 9 , wherein
the first reliability utilizes, as an index, a reprojection error obtained when the one or more three-dimensional points included in the first three-dimensional model are projected onto an image plane of the one or more stationary cameras included in the first three-dimensional model, using the first camera parameters of the one or more stationary cameras, and
the second reliability utilizes, as an index, a reprojection error obtained when the one or more three-dimensional points included in the second three-dimensional model are projected onto an image plane of the one or more stationary cameras included in the second three-dimensional model, using the second camera parameters of the one or more stationary cameras.
11. The processing method according to claim 9 , wherein
in the matching:
when the third reliability is higher than a reference value, a first coordinate matching scheme which matches the first world coordinate system of the first three-dimensional model and the second world coordinate system of the second three-dimensional model is selected based on an error between (i) part of the first camera parameters of the one or more stationary cameras and the one or more stationary three-dimensional points included in the first three-dimensional model and (ii) part of the second camera parameters of the one or more stationary cameras and the one or more stationary three-dimensional points included in the second three-dimensional model; and
when the third reliability is lower than the reference value, a second coordinate matching scheme which matches the first world coordinate system of the first three-dimensional model and the second world coordinate system of the second three-dimensional model is selected based on an error between (i) all of the first camera parameters of the one or more stationary cameras and the one or more stationary three-dimensional points included in the first three-dimensional model and (ii) all of the second camera parameters of the one or more stationary cameras and the one or more stationary three-dimensional points included in the second three-dimensional model.
12. A three-dimensional model processing apparatus, comprising:
a processor; and
memory, wherein
using the memory, the processor:
generates a first three-dimensional model from first images shot by respective cameras at a first time, the first three-dimensional model including: first three-dimensional points indicating a subject at the first time; and first camera parameters indicating positions and orientations of the cameras;
generates a second three-dimensional model from second images shot by the respective cameras at a second time different from the first time, the second three-dimensional model including: second three-dimensional points indicating the subject at the second time; and second camera parameters indicating positions and orientations of the cameras;
detects (1) one or more stationary cameras among the cameras, positions and orientations of the one or more stationary cameras not having changed between the first time and the second time, or (2) one or more stationary three-dimensional points among the plurality of three-dimensional points, positions of the one or more stationary three-dimensional points not having changed between the first time and the second time; and
matches a first world coordinate system of the first three-dimensional model and a second world coordinate system of the second three-dimensional model, based on the first camera parameters of the one or more stationary cameras or the one or more stationary three-dimensional points detected.