Estimating a pose of a camera for volume estimation
What is disclosed a system and method for estimating a position (or pose) of a camera relative to a surface upon which an object rests in an image captured by that camera such that a volume can be estimated for that object. In one embodiment, a matrix K is determined from parameters intrinsic to a camera used to capture image. An amount of a camera translation T is determined with respect to a set of real-world coordinates in (X,Y,Z). An amount of a camera rotation matrix R is determined from camera angles measured with respect to the real-world coordinates. A distance Z c of the camera at location (i,j) can then be estimated. A volume of the object in an image of that object can be estimated from the camera pose.
1. A method for estimating a pose of a camera relative to a surface upon which a 3D object rests such that a volume can be estimated for that object in an image captured by that camera, the method comprising:
determining a matrix K from parameters intrinsic to a camera used to capture at least one image of an object for volume estimation purposes;
determining a camera translation vector T=[t X t Y t Z ] relative to a set of real-world reference coordinates in (X,Y,Z);
determining a camera rotation matrix R from camera angles (φ,θ,ω) relative to the X, Y and Z axis, respectively, of said real-world reference coordinates, wherein said camera translation and said camera rotation are determined from depth values associated with pixels of a surface of said object, said depth values being measured by a depth sensor of said camera; and
estimating a distance Z c of said camera, comprising:
Z
c
[
i
j
1
]
=
KR
[
X
Y
Z
]
+
KT
T
wherein K comprises:
K
=
[
f
1
α
f
1
c
1
0
f
2
c
2
0
0
1
]
,
where f 1 and f 2 are focal lengths of a lens of said camera in each of two directions, α is a non-zero skew of a sensor of said camera, and c 1 and c 2 are center positions of said camera in pixels,
and; wherein R comprises:
R
=
[
1
0
0
0
cos
φ
-
sin
φ
0
sin
φ
cos
φ
]
[
cos
θ
0
-
sin
θ
0
1
0
sin
θ
0
cos
θ
]
[
cos
ω
-
sin
ω
0
sin
ω
cos
ω
0
0
0
1
]
.
2. The system of claim 1 , wherein a 2-dimensional surface model is fitted to said measured depth values of said surface.
3. The method of claim 1 , further comprising providing instructions to an operator of said camera to rotate said camera so that a rotation thereof approximates a rotation angle any of: (90°, 90°, 0°), (90°, 0°, 90°), and (0°, 90°, 90°), said instructions being based on a depth distribution of said surface.
4. The method of claim 1 , further comprising computing the real-world coordinates XYZ of the surface at each pixel location (i,j), X S (i,j),Y S (i,j),Z S (i,j), or the real-world coordinates XYZ of an object rested on said surface at each pixel location (i,j), X O (i,j), Y O (i,j),Z O (i,j), using said equation in claim 1 and the depth output d S (i,j) or d O (i,j) from the depth sensor of said surface or said object in an image captured by said camera.
5. The method of claim 1 , further comprising estimating a volume V of an object in an image captured by said camera, said volume comprising:
V
=
∑
(
i
,
j
)
∈
Ω
Δ
Z
(
i
,
j
)
·
Δ
X
(
i
,
j
)
·
Δ
Y
(
i
,
j
)
,
where Ω is a set of pixel locations of object in said image; ΔZ(i,j)=Z s (i,j)−Z o (i,j) is an angle-corrected delta between depth of surface of object d o (i,j) at location (i,j) as measured by a sensor and depth of surface d s (i,j) at the (i,j) position as determined by the aforementioned plane fitting; and ΔX(i,j)=X o (i,j)−X o (i−1,j), ΔY(i,j)=Y o (i,j)−Y o (i,j−1) are physical sizes corresponding to an area imaged by pixel (i,j) in real-world coordinates obtained by calculating deltas along X and Y axes between pixel (i,j) and its neighboring pixels, angle correction delta ΔZ(i,j) is a function of cos φ, cos θ, cos ω and d s (i,j)−d o (i,j).
6. The method of claim 1 , further comprising:
retrieving a model of said object from a database; and
using said model to refine said volume estimation.
7. The method of claim 1 , further comprising:
receiving a user input as to a type of said object; and
using said input to refine said volume estimation.
8. The method of claim 7 , wherein said object is a food item, further comprising estimating a volume of said food item.
9. A system for estimating a pose of a camera relative to a surface upon which a 3D object rests such that a volume can be estimated for that object in an image captured by that camera, said system comprising:
a memory; and
a processor in communication with said memory, said processor executing machine readable program instructions for performing the method of:
determining a matrix K from parameters intrinsic to a camera used to capture at least one image of an object for volume estimation purposes;
determining a camera translation vector T=[t X t Y t Z ] relative to a set of real-world reference coordinates in (X,Y,Z);
determining a camera rotation matrix R from camera angles (φ,θ,ω) relative to the X, Y and Z axis, respectively, of said real-world reference coordinates, wherein said camera translation and said camera rotation are determined from depth values associated with pixels of a surface of said object, said depth values being measured by a depth sensor of said camera; and
estimating a distance Z c of said camera, comprising:
Z
c
[
i
j
1
]
=
KR
[
X
Y
Z
]
+
KT
T
wherein K comprises:
K
=
[
f
1
α
f
1
c
1
0
f
2
c
2
0
0
1
]
,
where f 1 and f 2 are focal lengths of a lens of said camera in each of two directions, α is a non-zero skew of a sensor of said camera, and c 1 and c 2 are center positions of said camera in pixels,
and; wherein R comprises:
R
=
[
1
0
0
0
cos
φ
-
sin
φ
0
sin
φ
cos
φ
]
[
cos
θ
0
-
sin
θ
0
1
0
sin
θ
0
cos
θ
]
[
cos
ω
-
sin
ω
0
sin
ω
cos
ω
0
0
0
1
]
.
10. The system of claim 9 , wherein a 2-dimensional surface model is fitted to said measured depth values of said surface.
11. The system of claim 9 , further comprising providing instructions to an operator of said camera to rotate said camera so that a rotation thereof approximates a rotation angle any of: (90°, 90°, 0°), (90°, 0°, 90°), and (0°, 90°, 90°), said instructions being based on a depth distribution of said surface.
12. The system of claim 9 , further comprising computing the real-world coordinates XYZ of the surface at each pixel location (i,j), X S (i,j),Y S (i,j),Z S (i,j), or the real-world coordinates XYZ of an object rested on said surface at each pixel location (i,j), X O (i,j), Y O (i,j),Z O (i,j), using said equation in claim 1 and the depth output d S (i,j) or d O (i,j) from the depth sensor of said surface or said object in an image captured by said camera.
13. The system of claim 9 , further comprising estimating a volume V of an object in an image captured by said camera, said volume comprising:
V
=
∑
(
i
,
j
)
∈
Ω
Δ
Z
(
i
,
j
)
·
Δ
X
(
i
,
j
)
·
Δ
Y
(
i
,
j
)
,
where Ω is a set of pixel locations of object in said image; ΔZ(i,j)=Z s (i,j)−Z o (i,j) is an angle-corrected delta between depth of surface of object d o (i,j) at location (i,j) as measured by a sensor and depth of surface d s (i,j) at the (i,j) position as determined by the aforementioned plane fitting; and ΔX(i,j)=X o (i,j)−X o (i−1,j), ΔY(i,j)=Y o (i,j)−Y o (i,j−1) are physical sizes corresponding to an area imaged by pixel (i,j) in real-world coordinates obtained by calculating deltas along X and Y axes between pixel (i,j) and its neighboring pixels, angle correction delta ΔZ(i,j) is a function of cos φ, cos θ, cos ω and d s (i,j)−d o (i,j).
14. The system of claim 9 , further comprising:
retrieving a model of said object from a database; and
using said model to refine said volume estimation.
15. The system of claim 9 , further comprising:
receiving a user input as to a type of said object; and
using said input to refine said volume estimation.
16. The system of claim 15 , wherein said object is a food item, further comprising estimating a volume of said food item.