Aligning sequences by generating encoded representations of data items
An encoder neural network is described which can encode a data item, such as a frame of a video, to form a respective encoded data item. Data items of a first data sequence are associated with respective data items of a second sequence, by determining which of the encoded data items of the second sequence is closest to the encoded data item produced from each data item of the first sequence. Thus, the two data sequences are aligned. The encoder neural network is trained automatically using a training set of data sequences, by an iterative process of successively increasing cycle consistency between pairs of the data sequences.
1 . A method of aligning two video sequences of events using a single encoder neural network, a first video sequence of the two video sequences being a sequence of first video frames each comprising image data captured by at least one video camera, and a second video sequence of the two video sequences being a sequence of second video frames each comprising image data captured by at least one video camera, the sequence of first video frames depicting respective first events which occur in a first environment at successive first times, and the sequence of second video frames depicting respective second events which occur in a second environment at successive second times, the first video sequence depicting a first task and the second video sequence depicting a second task, the first task and the second task each having a plurality of corresponding phases that are performed in the same order in the first and second video sequences, the method comprising the steps of:
encoding the first video sequence using an encoder neural network to form, from each first video frame of the sequence of first video frames, a corresponding first encoded video frame;
attributing annotation data to one or more first video frames of the first video sequence, wherein the annotation data specifies a phase of the first task performed in an environment to which a video frame to which it is attributed relates;
encoding the second video sequence using the encoder neural network to form, from each second video frame of the sequence of second video frames, a corresponding second encoded video frame; and
for at least one first video frame of the sequence of first video frames:
(i) for each of a plurality of the second video frames, determining a respective distance value indicative of a distance between the corresponding first encoded video frame and the corresponding second encoded video frame according to a distance measure;
(ii) determining one of the plurality of second video frames for which the corresponding distance value is lowest;
(iii) associating the first video frame and the determined one of the plurality of second video frames, to associate the corresponding first event with the corresponding second event, the first and second events being events of a same type;
(iv) based on the association of the first event and the second event, associating the annotation data with second video frames of the second video sequence; and
(v) using the annotation data associated with the second video frames of the second video sequence as an input to a control program to determine that a first phase of the second task has been completed and to initiate a second phase of the second task.
2 . A method according to claim 1 in which the steps are performed while the sequence of first video frames of the first video sequence are datasets successively captured by at least one sensor and characterizing a real world environment at successive times.
3 . A method according to claim 2 , wherein using the annotation data associated with the second video frames of the second video sequence as an input to a control program comprises: in response to capturing a video frame of the first video sequence, and associating the first video frame and the determined one of the plurality of second video frames,
identifying the annotation data associated with the determined one of the plurality of second video frames, using the annotation data to generate control signals, and based on the control signals, modifying the real world environment.
4 . A method according to claim 1 further comprising determining whether one or more of the distance values meet an anomaly criterion, and if the anomaly criterion is met transmitting a warning message.
5 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for aligning two video sequences of events using a single encoder neural network, a first video sequence of the two video sequences being a sequence of first video frames each comprising image data captured by at least one video camera, and a second video sequence of the two video sequences being a sequence of second video frames each comprising image data captured by at least one video camera, the sequence of first video frames depicting respective first events which occur in a first environment at successive first times, and the sequence of second video frames depicting respective second events which occur in a second environment at successive second times, the first video sequence depicting a first task and the second video sequence depicting a second task, the first task and the second task each having a plurality of corresponding phases that are performed in the same order in the first and second video sequences, the operations comprising:
encoding the first video sequence using an encoder neural network to form, from each first video frame of the sequence of first video frames, a corresponding first encoded video frame;
attributing annotation data to one or more first video frames of the first video sequence, wherein the annotation data specifies a phase of the first task performed in an environment to which a video frame to which it is attributed relates;
encoding the second video sequence using the encoder neural network to form, from each second video frame of the sequence of second video frames, a corresponding second encoded video frame; and
for at least one first video frame of the sequence of first video frames:
(i) for each of a plurality of the second video frames, determining a respective distance value indicative of a distance between the corresponding first encoded video frame and the corresponding second encoded video frame according to a distance measure;
(ii) determining one of the plurality of second video frames for which the corresponding distance value is lowest;
(iii) associating the first video frame and the determined one of the plurality of second video frames, to associate the corresponding first event with the corresponding second event, the first and second events being events of a same type;
(iv) based on the association of the first event and the second event, associating the annotation data with second video frames of the second video sequence; and
(v) using the annotation data associated with the second video frames of the second video sequence as an input to a control program to determine that a first phase of the second task has been completed and to initiate a second phase of the second task.
6 . A system according to claim 5 in which the operations are performed while the sequence of first video frames of the first video sequence are datasets successively captured by at least one sensor and characterizing a real world environment at successive times.
7 . A system according to claim 6 wherein using the annotation data associated with the second video frames of the second video sequence as an input to a control program comprises: in response to capturing a video frame of the first video sequence, and associating the first video frame and the determined one of the plurality of second video frames,
identifying the annotation data associated with the determined one of the plurality of second video frames, using the annotation data to generate control signals, and based on the control signals, modifying the real world environment.
8 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for aligning two video sequences of events using a single encoder neural network, a first video sequence of the two video sequences being a sequence of first video frames each comprising image data captured by at least one video camera, and a second video sequence of the two video sequences being a sequence of second video frames each comprising image data captured by at least one video camera, the sequence of first video frames depicting respective first events which occur in a first environment at successive first times, and the sequence of second video frames depicting respective second events which occur in a second environment at successive second times, the first video sequence depicting a first task and the second video sequence depicting a second task, the first task and the second task each having a plurality of corresponding phases that are performed in the same order in the first and second video sequences, the operations comprising:
encoding the first video sequence using an encoder neural network to form, from each first video frame of the sequence of first video frames, a corresponding first encoded video frame;
attributing annotation data to one or more first video frames of the first video sequence, wherein the annotation data specifies a phase of the first task performed in an environment to which a video frame to which it is attributed relates;
encoding the second video sequence using the encoder neural network to form, from each second video frame of the sequence of second video frames, a corresponding second encoded video frame; and
for at least one first video frame of the sequence of first video frames:
(i) for each of a plurality of the second video frames, determining a respective distance value indicative of a distance between the corresponding first encoded video frame and the corresponding second encoded video frame according to a distance measure;
(ii) determining one of the plurality of second video frames for which the corresponding distance value is lowest;
(iii) associating the first video frame and the determined one of the plurality of second video frames, to associate the corresponding first event with the corresponding second event, the first and second events being events of a same type;
(iv) based on the association of the first event and the second event, associating the annotation data with second video frames of the second video sequence; and
(v) using the annotation data associated with the second video frames of the second video sequence as an input to a control program to determine that a first phase of the second task has been completed and to initiate a second phase of the second task.
9 . One or more non-transitory computer storage media according to claim 8 in which the operations are performed while the sequence of first video frames of the first video sequence are datasets successively captured by at least one sensor and characterizing a real world environment at successive times.
10 . One or more non-transitory computer storage media according to claim 8 , wherein using the annotation data associated with the second video frames of the second video sequence as an input to a control program comprises: in response to capturing a video frame of the first video sequence, and associating the first video frame and the determined one of the plurality of second video frames,
identifying the annotation data associated with the determined one of the plurality
of second video frames, using the annotation data to generate control signals, and based on the control signals, modifying a real world environment.