Video monitoring using semantic segmentation based on global optimization
Methods and systems for monitoring a video feed include capturing input data representing a monitored area using one or more cameras to produce an input data set comprising individual images. Initial segmentation scores are determined for each image of the input data set using a neural network, with each image being assigned an initial segmentation score for each of multiple segmentation classes. Final segmentation scores are determined for each image of the input data set by enforcing a smoothness criterion. The input data set is segmented in accordance with the final segmentation scores. It is determined whether an alert condition is met based on segmented data set. An alert is generated if the alert condition is met.
1. A method for monitoring a video feed, comprising:
capturing input data representing a monitored area using one or more cameras to produce an input data set comprising individual images;
determining initial segmentation scores for each image of the input data set using a neural network, with each image being assigned an initial segmentation score for each of a plurality of segmentation classes;
determining final segmentation scores for each image of the input data set by enforcing a smoothness criterion;
segmenting the input data set in accordance with the final segmentation scores,
wherein the determining final segmentation scores comprises applying variational reaction diffusion (VRD),
wherein the applying VRD comprises solving an optimization problem based on the initial segmentation scores and on positive-definite parameter matrices, and
wherein the optimization problem is expressed as:
arg
min
s
o
∫
I
s
T
Qs
+
∑
k
=
1
2
∂
s
T
∂
x
k
B
∂
s
∂
x
k
dx
where s o is a vector of the final segmentation scores for a unit of data, x is a unit of data in data set I, d is a dimension of the data set I, and s is a vector combining s o and a vector of the initial segmentation scores for the unit of data; and
backpropagating information for efficiently training the neural network using the final segmentation scores and an error signal, wherein computational complexity for the training is reduced by reducing computations for the training to the same steps for generating the final segmentation scores;
determining if an alert condition is met based on the segmented input data set; and
generating an alert if the alert condition is met.
2. The method of claim 1 , wherein solving the optimization problem comprises solving a system of partial differential equations.
3. The method of claim 2 , wherein applying VRD comprises converting the optimization problem to the system of partial differential equations using Schur decomposition on the positive-definite parameter matrices.
4. The method of claim 2 , wherein the system of partial differential equations is expressed as:
Δ
z
k
-
U
kk
z
k
=
(
V
T
(
B
o
)
-
1
s
p
)
k
+
∑
j
=
k
+
1
N
o
U
kj
z
j
where s p is defined as the expression Q i s i −B i Δs i , Q i , Q o , B i , and B o are sub-matrices of the positive-definite parameter matrices, Δ is the vector Laplacian operator, s i is the initial segmentation scores for an image, each z k represents a transformed score on a k th segmentation class, N o is a number of segmentation classes, and V and U are transformed versions of the positive-definite parameter matrices after Schur decomposition.
5. The method of claim 1 , wherein backpropagating information comprises determining derivatives of the error signal with respect to the positive-definite parameter matrices.
6. The method of claim 1 , wherein the alert condition comprises a condition selected from the set of a final segmentation score for a predetermined one of the plurality of segmentation classes exceeding a threshold and a change in the final segmentation scores that exceeds a threshold.
7. A system for data segmentation, comprising:
one or more cameras configured to produce an input data set comprising individual images;
a neural network configured to determine initial segmentation scores for each image of an input data set, with each image being assigned an initial segmentation score for each of a plurality of segmentation classes;
a segmentation module configured to determine final segmentation scores for each image of the input data set by enforcing a smoothness criterion and to segment the input data set in accordance with the final segmentation scores,
wherein the segmentation module is further configured to apply variational reaction diffusion (VRD) to determine the final segmentation scores,
wherein the segmentation module is further configured to solve a VRD optimization problem based on the initial segmentation scores and on positive-definite parameter matrices,
wherein the optimization problem is expressed as:
arg
min
s
o
∫
I
s
T
Qs
+
∑
k
=
1
2
∂
s
T
∂
x
k
B
∂
s
∂
x
k
dx
where s o is a vector of the final segmentation scores for a unit of data, x is a unit of data in data set I, d is a dimension of the data set I, and s is a vector combining s o and a vector of the initial segmentation scores for the unit of data; and
wherein the neural network is further configured to backpropagate information for training the neural network using the final segmentation scores and an error signal, wherein computational complexity for the training is reduced by reducing computations for the training to the same steps for generating the final segmentation scores; and
an alert module configured to determine if an alert condition is met based on the segmented input data set and to generate an alert if the alert condition is met.
8. The system of claim 7 , wherein the segmentation module is further configured to solve a system of partial differential equations to solve the optimization problem.
9. The system of claim 8 , wherein the segmentation module is further configured to convert the optimization problem to the system of partial differential equations using Schur decomposition on the positive-definite parameter matrices.
10. The system of claim 8 , wherein the system of partial differential equations is expressed as:
Δ
z
k
-
U
kk
z
k
=
(
V
T
(
B
o
)
-
1
s
p
)
k
+
∑
j
=
k
+
1
N
o
U
kj
z
j
where s p is defined as the expression Q i s i −B i Δs i , Q i , Q o , B i , and B o are sub-matrices of the positive-definite parameter matrices, Δ is the vector Laplacian operator, s i is the initial segmentation scores for a unit of data, each z k represents a transformed score on a k th segmentation class, N o is a number of segmentation classes, and V and U are transformed versions of the positive-definite parameter matrices after Schur decomposition.
11. The system of claim 7 , wherein the neural network is further configured to determine derivatives of the error signal with respect to the positive-definite parameter matrices.
12. The system of claim 7 , wherein the alert condition comprises a condition selected from the set of a final segmentation score for a predetermined one of the plurality of segmentation classes exceeding a threshold and a change in the final segmentation scores that exceeds a threshold.