System and method for generation of unseen composite data objects
A computer implemented system for generating one or more data structures is described, the one or more data structures representing an unseen composition based on a first category and a second category observed individually in a training data set. During training of a generator, a proposed framework utilizes at least one of the following discriminators—three pixel-centric discriminators, namely, frame discriminator, gradient discriminator, video discriminator; and one object-centric relational discriminator. The three pixel-centric discriminators ensure spatial and temporal consistency across the frames, and the relational discriminator leverages spatio-temporal scene graphs to reason over the object layouts in videos ensuring the right interactions among objects.
1. A computer implemented system for generating one or more data structures, the one or more data structures representing an unseen composition based on a first category and a second category observed individually, the system comprising:
one or more processors operating in conjunction with computer memory, the one or more processors configured to provide:
a data receiver interface configured to receive a training data set including labelled data elements based on the first category and labelled data elements based on the second category and to receive a target category indication representative of the unseen composition;
a conditional generative adversarial network configured to process the training data set to train a discriminator model architecture coupled to a generator model architecture, the discriminator model architecture having a plurality of adversarial networks operating in concert to train the generator model architecture, the discriminator model architecture including at least:
a sequence discriminator configured to distinguish between a real sequence and a generated sequence;
a frame discriminator configured to differentiate between frames representing sequence subsets of the real sequence and the generated sequence;
a gradient discriminator configured to differentiate between a domain-specific gradient determined based on the type of data structure of the one or more data structures and the training data set; and
a relational discriminator configured to assign weights for shifting focus of the generator model architecture to a subset of the one or more new data structures based on an identified context associated with the target category indication of the unseen composition;
wherein the generator model architecture is configured to generate the one or more data structures representing the unseen composition based on the outputs of the plurality of adversarial networks.
2. The system of claim 1 , wherein the first category includes a set of actions, the second category includes a set of objects, and the training data set includes a plurality of data structures of action/object pairs different than the target category indication representative of the unseen composition.
3. The system of claim 2 , wherein the new data structures includes at least a new video data structure generated to represent an action/object pair representative of the unseen composition by synthesizing independently observed data represented in the training data set.
4. The system of claim 1 , wherein the first category includes vectorized transactional information and wherein the second category includes vectorized representation of one or more events.
5. The system of claim 1 , wherein vectorized labels associated with each training data element in the training data set are processed to identify one or more contextual components that are used for comparison with a vector representing the unseen composition, the one or more contextual components utilized for modifying the operation of the discriminator model architecture.
6. The system of claim 1 , wherein the sequence discriminator utilizes a loss function having the relation:
L v =½[log( D v ( V real ,s a ,s o )+log(1− D v ( V gen ,s a ,s o ))].
7. The system of claim 1 , wherein the frame discriminator utilizes a loss function having the relation:
L
f
=
1
2
T
∑
i
=
1
T
[
log
(
D
f
i
(
V
real
,
s
a
,
s
o
)
+
log
(
1
-
D
f
i
(
V
gen
,
s
a
,
s
o
)
)
]
.
8. The system of claim 1 , wherein the gradient discriminator utilizes a loss function having the relation:
L
g
=
1
2
(
T
-
1
)
∑
i
=
1
T
-
1
[
log
(
D
g
i
(
δ
V
real
,
s
a
,
s
o
)
+
log
(
1
-
D
g
i
(
δ
V
gen
,
s
a
,
s
o
)
)
]
.
9. The system of claim 1 , wherein the relational discriminator utilizes a loss function having the relation:
L
fg
=
1
2
T
∑
i
=
1
T
[
log
(
D
fg
i
(
F
real
,
s
a
,
s
o
)
+
log
(
1
-
D
fg
i
(
F
gen
,
s
a
,
s
o
)
)
]
.
10. The system of claim 1 , wherein the generator model architecture is configured to be optimized using an objective function having the relation:
ℒ
gan
=
log
(
1
-
D
v
(
V
gen
,
s
a
,
s
o
)
)
]
+
1
T
∑
i
=
1
T
[
log
(
1
-
D
f
i
(
V
gen
,
s
a
,
s
o
)
)
]
+
1
(
T
-
1
)
∑
i
=
1
T
-
1
[
log
(
1
-
D
g
i
(
δ
V
gen
,
s
a
,
s
o
)
)
]
+
1
T
∑
i
=
1
T
[
log
(
1
-
D
fg
i
(
F
gen
,
s
a
,
s
o
)
)
]
.
11. A computer implemented method for generating one or more data structures using a conditional generative adversarial network, the one or more data structures representing an unseen composition based on a first category and a second category observed individually, the method comprising:
receiving a training data set including labelled data elements based on the first category and labelled data elements based on the second category;
receiving a target category indication representative of the unseen composition;
processing the training data set to train a discriminator model architecture coupled to a generator model architecture, the discriminator model architecture including at least:
a relational discriminator D r configured to assign weights for shifting focus of the generator model architecture to a subset of the one or more new data structures based on an identified context associated with the target category indication of the unseen composition; and
generating using the generator model architecture the one or more data structures;
wherein the relational discriminator utilizes a spatio-temporal scene graph, and adapts a neural network to distinguish between element layouts of real data objects V real and generated data objects V gen ;
wherein the spatio-temporal scene graph is represented as =( , ε) and generated from V, where the nodes and edges are represented by and ε.
12. The method of claim 11 , wherein the relational discriminator operates on scene graph using a graph convolutional network (GCN) followed by stacking and average-pooling of the resulting node representations along the time axis.
13. The method of claim 12 , wherein the scene graph is the concatenated with spatially replicated copies of s a and s o to generate a tensor of size (dim(s a )+dim(s o )+N (t) )×w 0 (t) ×h 0 (t) , wherein s a and s o represent word embeddings of two different characteristics.
14. The method of claim 13 , the method further comprising applying convolutions and sigmoid to the tensor of size (dim(s a )+dim(s o )+N (t) )×w 0 (t) ×h 0 (t) to obtain an intermediate output which denotes the probability of the scene graph belonging to a real data object, the intermediate output used to assign the weights for shifting focus of the generator model architecture.
15. The method of claim 11 , wherein an objective function of the relational discriminator is given by:
L r =½[log( D r ( real ;s a ,s o ))+log(1− D r ( gen ;s a ,s o ))].
16. The method of claim 11 , wherein the discriminator model architecture further includes a sequence discriminator configured to distinguish between a real sequence and a generated sequence.
17. The method of claim 16 , wherein the discriminator model architecture further includes a gradient discriminator configured to differentiate between a domain-specific gradient determined based on the type of data structure of the one or more data structures and the training data set.
18. The method of claim 17 , wherein the discriminator model architecture further includes a frame discriminator configured to differentiate between frames representing sequence subsets of the real sequence and the generated sequence.
19. The method of claim 18 , wherein the relational discriminator, the sequence discriminator, the gradient discriminator, and the frame discriminator are trained simultaneously.
20. A non-transitory, computer readable medium, storing machine interpretable instructions, which when executed by a processor, cause the processor to perform a computer implemented method of generating one or more data structures using a conditional generative adversarial network, the one or more data structures representing an unseen composition based on a first category and a second category observed individually, the method comprising:
receiving a training data set including labelled data elements based on the first category and labelled data elements based on the second category;
receiving a target category indication representative of the unseen composition;
processing the training data set to train a discriminator model architecture coupled to a generator model architecture, the discriminator model architecture having a plurality of adversarial networks operating in concert to train the generator model architecture, the discriminator model architecture including at least:
a relational discriminator configured to assign weights for shifting focus of the generator model architecture to a subset of the one or more new data structures based on an identified context associated with the target category indication of the unseen composition; and
generating, using the generator model architecture, the one or more data structures;
wherein the relational discriminator utilizes a spatio-temporal scene graph, and learns to distinguish between element layouts of real element objects V real and generated data elements V gen ;
wherein the spatio-temporal scene graph is represented as =( , ε) and generated from V, where the nodes and edges are represented by and ε.