Granted Patent B2

US 12,592,027 · App. 18/251,224 · Granted Mar 31, 2026

Inventors: Zhili Chen (Los Angeles, CA); Linjie Luo (Los Angeles, CA)

Assignee: BYTEDANCE INC.

G06T17/00

Abstract

Claims (292)

1 . A three-dimensional reconstruction method, comprising:

obtaining a plurality of input images each comprising a target object, and obtaining coordinates of a plurality of first positions, in a geodetic coordinate system, of an image capturing device corresponding to the plurality of input images;

constructing a reconstruction coordinate system based on the plurality of input images, and performing a three-dimensional reconstruction processing for the target object based on the reconstruction coordinate system, to obtain a first three-dimensional reconstruction model of the target object and coordinates of a plurality of second positions, in the reconstruction coordinate system, of the image capturing device corresponding to the plurality of input images; and

performing a transformation processing on at least part of points in the first three-dimensional reconstruction model based on the coordinates of the plurality of first positions and the coordinates of the plurality of second positions to obtain a second three-dimensional reconstruction model.

2 . The three-dimensional reconstruction method according to claim 1 , wherein said performing the transformation processing on the at least part of points in the first three-dimensional reconstruction model based on the coordinates of the plurality of first positions and the coordinates of the plurality of second positions to obtain the second three-dimensional reconstruction model comprises:

calculating a transformation matrix between the reconstruction coordinate system and the geodetic coordinate system based on the coordinates of the plurality of first positions and the coordinates of the plurality of second positions; and

performing the transformation processing on the at least part of points in the first three-dimensional reconstruction model using the transformation matrix to obtain the second three-dimensional reconstruction model.

3 . The three-dimensional reconstruction method according to claim 2 , wherein said calculating the transformation matrix between the reconstruction coordinate system and the geodetic coordinate system based on the coordinates of the plurality of first positions and the coordinates of the plurality of second positions comprises:

transforming the coordinates of the plurality of first positions from the geodetic coordinate system to an intermediate coordinate system to obtain coordinates of a plurality of third positions, in the intermediate coordinate system, of the image capturing device corresponding to the plurality of input images, the coordinates of the plurality of third positions corresponding to the coordinates of the plurality of first positions in one-to-one correspondence, and said transforming from the geodetic coordinate system to the intermediate coordinate system being based on a first transformation sub-matrix;

calculating a second transformation sub-matrix between the intermediate coordinate system and the reconstruction coordinate system based on the coordinates of the plurality of second positions and the coordinates of the plurality of third positions; and

calculating the transformation matrix based on the first transformation sub-matrix and the second transformation sub-matrix.

4 . The three-dimensional reconstruction method according to claim 3 , wherein:

the second transformation sub-matrix is expressed as:

M

=

[

s

*

r

⁢

11

s

*

r

⁢

12

s

*

r

⁢

13

t

⁢

x

s

*

r

⁢

21

s

*

r

⁢

2

⁢

2

s

*

r

⁢

2

⁢

3

t

⁢

y

s

*

r

⁢

31

s

*

r

⁢

3

⁢

2

s

*

r

⁢

3

⁢

3

t

⁢

z

0

0

0

1

]

,

where M represents the second transformation sub-matrix, s represents a scaling factor, r11, r12, r13, r21, r22, r23, r31, r32, and r33 represent rotation parameters, and tx, ty, and tz represent translation parameters; and

the coordinates of the plurality of second positions, the coordinates of the plurality of third positions, and the second transformation sub-matrix satisfy a relationship of:

Pi=M*Qi,

where Pi represents a matrix corresponding to coordinates of an i-th third position in the coordinates of the plurality of third positions, and Qi represents a matrix corresponding to coordinates of an i-th second position in the coordinates of the plurality of second positions,

wherein Qi is expressed as:

Qi

=

[

x

⁢

1

⁢

i

y

⁢

1

⁢

i

z

⁢

1

⁢

i

1

]

,

where x1i, y1i, and z1i represent the coordinates of the i-th second position, and

Pi is expressed as:

Pi

=

[

x

⁢

2

⁢

i

y

⁢

2

⁢

i

z

⁢

2

⁢

i

1

]

,

where x2i, y2i, and z2i represent the coordinates of the i-th third position.

5 . The three-dimensional reconstruction method according to claim 3 , wherein the intermediate coordinate system is a local Cartesian coordinates coordinate system, an origin of the local Cartesian coordinates coordinate system being average values of the coordinates of the plurality of first positions.

6 . The three-dimensional reconstruction method according to claim 1 , wherein said performing the transformation processing on the at least part of points in the first three-dimensional reconstruction model based on the coordinates of the plurality of first positions and the coordinates of the plurality of second positions to obtain the second three-dimensional reconstruction model comprises:

transforming the coordinates of the plurality of first positions from the geodetic coordinate system to an intermediate coordinate system to obtain coordinates of a plurality of third positions, in the intermediate coordinate system, of the image capturing device corresponding to the plurality of input images, the coordinates of the plurality of third positions corresponding to the coordinates of the plurality of first positions in one-to-one correspondence, and said transforming from the geodetic coordinate system to the intermediate coordinate system being based on a first transformation sub-matrix;

calculating a second transformation sub-matrix between the intermediate coordinate system and the reconstruction coordinate system based on the coordinates of the plurality of second positions and the coordinates of the plurality of third positions, as the transformation matrix; and

performing the transformation processing on the at least part of points in the first three-dimensional reconstruction model using the transformation matrix to obtain the second three-dimensional reconstruction model.

7 . The three-dimensional reconstruction method according to claim 6 , wherein:

the second transformation sub-matrix is expressed as:

M

=

[

s

*

r

⁢

11

s

*

r

⁢

12

s

*

r

⁢

13

t

⁢

x

s

*

r

⁢

21

s

*

r

⁢

2

⁢

2

s

*

r

⁢

2

⁢

3

t

⁢

y

s

*

r

⁢

31

s

*

r

⁢

3

⁢

2

s

*

r

⁢

3

⁢

3

t

⁢

z

0

0

0

1

]

,

where M represents the second transformation sub-matrix, s represents a scaling factor, r11, r12, r13, r21, r22, r23, r31, r32, and r33 represent rotation parameters, and tx, ty, and tz represent translation parameters; and

the coordinates of the plurality of second positions, the coordinates of the plurality of third positions, and the second transformation sub-matrix satisfy a relationship of:

Pi=M*Qi,

where Pi represents a matrix corresponding to coordinates of an i-th third position in the coordinates of the plurality of third positions, and Qi represents a matrix corresponding to coordinates of an i-th second position in the coordinates of the plurality of second positions,

wherein Qi is expressed as:

Qi

=

[

x

⁢

1

⁢

i

y

⁢

1

⁢

i

z

⁢

1

⁢

i

1

]

,

where x1i, y1i, and z1i represent the coordinates of the i-th second position, and Pi is expressed as:

P

⁢

i

=

[

x

⁢

2

⁢

i

y

⁢

2

⁢

i

z

⁢

2

⁢

i

l

]

,

where x2i, y2i, and z2i represent the coordinates of the i-th third position.

8 . The three-dimensional reconstruction method according to claim 6 , wherein the intermediate coordinate system is a local Cartesian coordinates coordinate system, an origin of the local Cartesian coordinates coordinate system being average values of the coordinates of the plurality of first positions.

9 . The three-dimensional reconstruction method according to claim 1 , wherein said obtaining the coordinates of the plurality of first positions, in the geodetic coordinate system, of the image capturing device corresponding to the plurality of input images comprises:

reading a plurality of pieces of position data, which are recorded by the image capturing device at respective times when capturing the plurality of input images, and obtaining the coordinates of the plurality of first positions based on the plurality of pieces of position data.

10 . The three-dimensional reconstruction method according to claim 9 , wherein the plurality of pieces of position data is determined by a satellite positioning system.

11 . The three-dimensional reconstruction method according to claim 1 , wherein said constructing the reconstruction coordinate system based on the plurality of input images comprises:

determining a reference input image in the plurality of input images, wherein a number of input images among the plurality of input images which match the reference input image is maximal; and

establishing the reconstruction coordinate system using coordinates of a first position corresponding to the reference input image as a coordinate origin, a first reference direction as a first coordinate axis, a second reference direction as a second coordinate axis, and a view direction corresponding to the reference input image as a third coordinate axis.

12 . The three-dimensional reconstruction method according to claim 11 , wherein said performing the three-dimensional reconstruction processing for the target object based on the reconstruction coordinate system, to obtain the first three-dimensional reconstruction model of the target object and the coordinates of the plurality of second positions, in the reconstruction coordinate system, of the image capturing device corresponding to the plurality of input images comprises:

performing, on the plurality of input images, the three-dimensional reconstruction processing for the target object using a three-dimensional reconstruction algorithm to obtain the first three-dimensional reconstruction model;

determining coordinates of a second position corresponding to the reference input image as an origin of the reconstruction coordinate system; and

determining coordinates of second positions corresponding to remaining input images in the plurality of input images other than the reference input image based on position relationships between the remaining input images and the reference input image,

wherein the coordinates of the plurality of second positions comprise the coordinates of the second position corresponding to the reference input image and the coordinates of the second positions corresponding to the remaining input images.

13 . The three-dimensional reconstruction method according to claim 11 , wherein said performing the three-dimensional reconstruction processing for the target object based on the reconstruction coordinate system, to obtain the first three-dimensional reconstruction model of the target object and the coordinates of the plurality of second positions, in the reconstruction coordinate system, of the image capturing device corresponding to the plurality of input images comprises:

performing, on the plurality of input images, the three-dimensional reconstruction processing for the target object using a three-dimensional reconstruction algorithm to obtain an intermediate three-dimensional reconstruction model;

normalizing the intermediate three-dimensional reconstruction model to obtain the first three-dimensional reconstruction model;

determining coordinates of a second position corresponding to the reference input image as an origin of the reconstruction coordinate system; and

determining coordinates of second positions corresponding to remaining input images other than the reference input image in the plurality of input images based on position relationships between the remaining input images and the reference input image,

wherein the first three-dimensional reconstruction model is located within a normalized cube region in the reconstruction coordinate system, the normalized cube region ranging from −1 to 1 on each of the first coordinate axis, the second coordinate axis, and the third coordinate axis of the reconstruction coordinate system, and wherein the coordinates of the plurality of second positions comprise the coordinates of the second position corresponding to the reference input image and the coordinates of the second positions corresponding to the remaining input images.

14 . The three-dimensional reconstruction method according to claim 11 , wherein the first reference direction is a horizontal direction, and wherein the second reference direction is a vertical direction.

15 . The three-dimensional reconstruction method according to claim 1 , further comprising:

determining an actual scale of at least part of the target object based on the second three-dimensional reconstruction model.

16 . The three-dimensional reconstruction method according to claim 1 , wherein the first three-dimensional reconstruction model is a three-dimensional point cloud model.

17 . The three-dimensional reconstruction method according to claim 1 , wherein the plurality of first positions has different coordinates from each other, and coordinates of each of the plurality of first positions are at a different angle relative to the target object from coordinates of another of the plurality of first positions,

the coordinates of each of the plurality of first positions comprising a longitude, a latitude, and an altitude.

18 . The three-dimensional reconstruction method according to claim 1 , wherein the target object comprises a landmark building.

19 . A three-dimensional reconstruction apparatus, comprising:

a memory configured to store computer-readable instructions in a non-transitory manner; and

a processor configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, implement the three-dimensional reconstruction method according to claim 1 .

20 . A non-transitory computer-readable storage medium, having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the three-dimensional reconstruction method according to claim 1 .

Priority Claims (1)

Continuity (1)

References Cited (31)