Systems and methods for determining image representations at a pixel level
View Patent ↗Systems and methods process an image having a plurality of pixels includes an image sensor to capture an image; a first-layer to encode local patches on an image region; and a second layer to jointly encode patches from the same image region.
1. A method to process an image having a plurality of pixels, comprising:
capturing an image using an image sensor;
forming a first-layer to encode local patches on an image region;
forming a second layer to jointly encode patches from the image region; and
determining
1
n
X
-
BW
F
2
+
λ
2
n
∑
s
,
t
tr
(
S
(
W
(
s
,
t
)
)
Ω
(
s
,
t
)
(
α
)
)
where
Ω
(
s
,
t
)
(
α
)
≡
(
∑
k
=
1
q
α
k
(
ϕ
k
(
s
,
t
)
)
)
-
1
is an inverse diagonal covariance for an (s,t)-th region, s=1,2, t=1,2, X is a set of image patches in a local region and X and W denote all 4×n patches and their first-layer codes in these 4 regions for a fixed α, φ is a basis describing a a spatial co-occurrence pattern, B is a codebook, log-likelihood of W is tr (S(W)Ω(α)), λ 2 is a sparsely regulation parameter.
2. The method of claim 1 , comprising learning higher-order dependencies among related image patterns.
3. The method of claim 2 , comprising generating sparse, invariant, and discriminative image representations from the pixels.
4. The method of claim 1 , comprising learning the two-layer codebooks from unlabeled data.
5. The method of claim 1 , wherein the first layer encodes local patches of an image, whose energy pooled in a local region is passed to the second layer to jointly encode signals from the region.
6. The method of claim 1 , comprising learning high-order dependency among patterns in a local neighborhood of images.
7. The method of claim 1 , comprising modeling of spatial dependence.
8. The method of claim 1 , comprising performing hierarchical convolution coding.
9. The method of claim 1 , wherein the second layer forms a joint representation that models a covariance of the patches from neighboring image regions.
10. The method of claim 1 , comprising performing unsupervised codebook learning process with sparse coding.
11. The method of claim 10 , comprising receiving an initial codebook matrix and a sparsely regulation parameter; iteratively determining a sparse coding coefficient matrix; and updating the codebook matrix.
12. The method of claim 10 , comprising testing the learned codebook of sparse coding.
13. The method of claim 11 , comprising generating SVM weight vectors; determining an SVM score; and generating a prediction.
14. The method of claim 13 , comprising:
receiving a first codebook B, σ, and first regularization parameters to apply to m patch sets, each with n patches;
determining sparse coding coefficients; re-optimizing σ;
based on a second codebook Φ and second regularization parameters, determining a set level representation α;
re-optimizing B and Φ ( 322 ); and
checking for convergence and if convergence is met, finalizing B and Φ.
15. The method of claim 13 , comprising testing the learned codebook with hierarchical sparse coding.
16. The method of claim 13 , comprising:
determining sparse coding coefficients;
determining a set level representation;
determining an SVM score is determined from SVM weights; and
generating a prediction.
17. The method of claim 1 , comprising performing unsupervised codebook learning with hierarchical sparse coding.
18. A system to process an image having a plurality of pixels, comprising:
an image sensor to capture an image;
a first-layer to encode local patches on an image region; and
a second layer to jointly encode patches from the same image region; and
determining with a processor
1
n
X
-
BW
F
2
+
λ
2
n
∑
s
,
t
tr
(
S
(
W
(
s
,
t
)
)
Ω
(
s
,
t
)
(
a
)
)
where
Ω
(
s
,
t
)
(
α
)
≡
(
∑
k
=
1
q
α
k
(
ϕ
k
(
s
,
t
)
)
)
-
1
is an inverse diagonal covariance for an (s,t)-th region, s=1,2, t=1,2, X is a set of image patches in a local region and X and W denote all 4×n patches and their first-layer codes in these 4 regions for a fixed α, φ is a basis describing a a spatial co-occurrence pattern, B is a codebook, log-likelihood of W is tr (S(W)Ω(α)), λ 2 is sparsity regulation parameter.
19. The system of claim 18 , comprising means for learning the two-layer codebooks from unlabeled data.