Alignment of video and textual sequences for metadata analysis

1. A method of aligning heterogeneous sequential data, comprising:

receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments;

determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising:

a fully connected layer that receives as input:

data from the first sequential data stream,

data from the second sequential data stream, and

data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine the first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and

aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.

2. The method of claim 1 , wherein the fully connected layer further receives as input:

data relating to a previously determined match between the first sequential data stream and the second sequential data stream.

3. The method of claim 1 , wherein the first sequential data stream comprises video data and wherein the second sequential data stream comprises text data.

4. The method of claim 1 , wherein the alignment neural network further comprises a second fully connected layer that receives as input data from the fully connected layer.

5. The method of claim 1 , further comprising storing data related to the determined first alignment action.

6. The method of claim 1 , wherein the first alignment action comprises at least one of: a pop action related to the first sequential data stream, a pop action related to the second sequential data stream, a match action, or a match-retain action.

7. The method of claim 6 , wherein the first alignment action comprises the match action, and the method further comprises storing data related to the match action.

8. The method of claim 1 , further comprising:

receiving a third sequential data stream, wherein the determined first alignment action relates to aligning the first sequential data stream, the second sequential data stream, and the third sequential data stream.

9. The method of claim 8 , wherein the determined first alignment action is a parameterized action.

10. A computer program product for aligning heterogeneous sequential data, the computer program product comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising computer-readable program code configured to perform an operation, the operation comprising:

receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments;

determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising:

a fully connected layer that receives as input:

data from the first sequential data stream,

data from the second sequential data stream, and

data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine the first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and

aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.

11. The computer program product of claim 10 wherein the fully connected layer further receives as input:

data relating to a previously determined match between the first sequential data stream and the second sequential data stream.

12. The computer program product of claim 10 , wherein the first sequential data stream comprises video data and wherein the second sequential data stream comprises text data.

13. The computer program product of claim 10 , wherein the alignment neural network further comprises a second fully connected layer that receives as input data from the fully connected layer.

14. The computer program product of claim 10 , wherein the first alignment action comprises at least one of: a pop action related to the first sequential data stream, a pop action related to the second sequential data stream, a match action, or a match-retain action.

15. The computer program product of claim 10 , the operation further comprising: receiving a third sequential data stream, wherein the determined first alignment action relates to aligning the first sequential data stream, the second sequential data stream, and the third sequential data stream, and wherein the determined first alignment action is a parameterized action.

16. A system, comprising:

a processor; and

a memory containing a program that, when executed on the processor, performs an operation, the operation comprising:

receiving a first sequential data stream comprising a first plurality of segments and a second sequential data stream comprising a second plurality of segments;

determining a first alignment action related to aligning the first plurality of segments in the first sequential data stream with the second plurality of segments in the second sequential data stream using an alignment neural network, the alignment neural network comprising:

a fully connected layer that receives as input:

data from the first sequential data stream,

data from the second sequential data stream, and

data, retrieved from storage, relating to a plurality of previously determined alignment actions by the alignment neural network related to aligning the first sequential data stream and the second sequential data stream, wherein the alignment neural network is configured to determine a first alignment action based, at least in part, on the data related to the plurality of previously determined alignment actions; and

aligning a first segment in the first plurality of segments with a second segment in the second plurality of segments by performing the determined first alignment action.

17. The system of claim 16 , wherein the fully connected layer further receives as input:

data relating to a previously determined match between the first sequential data stream and the second sequential data stream.

18. The system of claim 16 , wherein the alignment neural network further comprises a second fully connected layer that receives as input data from the fully connected layer.

19. The system of claim 16 , wherein the first alignment action comprises a match action, and the operation further comprises storing data related to the match action.

20. The system of claim 16 , the operation further comprising:

receiving a third sequential data stream, wherein the determined first alignment action relates to aligning the first sequential data stream, the second sequential data stream, and the third sequential data stream, and wherein the determined first alignment action is a parameterized action.