Synchronization of multi-device image data using multimodal sensor data

1. A system, comprising:

circuitry configured to:

receive, from a plurality of image-capture devices, image data comprising a plurality of image sequences of a first object, wherein each image sequence of the plurality of image sequences corresponds to an image-capture device of the plurality of image-capture devices;

receive a set of sensor data from the plurality of image-capture devices,

wherein each sensor data of the received set of sensor data comprises at least one of an Inertial Measurement Unit (IMU) data or audio data, and

each sensor data of the received set of sensor data is associated with a duration of acquisition of a corresponding image sequence of the plurality of image sequences;

extract, from the received set of sensor data, a first IMU data corresponding to a first image sequence of the plurality of image sequences;

generate a first spectrogram of the first IMU data;

filter the generated first spectrogram based on one of a first two-dimensional (2D) diamond kernel or a first masked max filter to generate a first filter result;

convert the first filter result to a first list of one of time-domain values or frequency-domain values;

generate a first lookup key with first offset values to neighboring list elements of the first list;

determine, based on the generated first lookup key, a match between a first set of image frames of the first image sequence and a second set of image frames of a second image sequence of the plurality of image sequences;

compute an offset between the first set of image frames and the second set of image frames, based on the match; and

synchronize the first image sequence with the second image sequence based on the computed offset.

2. The system according to claim 1 , wherein the circuitry is further configured to control the plurality of image-capture devices to acquire the plurality of image sequences from a corresponding plurality of viewpoints.

3. The system according to claim 1 , further comprising a multi-camera rig which includes a support structure on which the plurality of image-capture devices is mounted.

4. The system according to claim 1 , wherein the circuitry is further configured to:

extract, from the received set of sensor data, a second IMU data corresponding to the second image sequence;

generate a second spectrogram of the second IMU data;

filter the generated second spectrogram based on one of the first 2D diamond kernel or the first masked max filter to generate a second filter result;

convert the second filter result to a second list of one of the time-domain values or the frequency-domain values; and

generate a second lookup key with second offset values to neighboring list elements of the second list.

5. The system according to claim 4 , wherein the circuitry is further configured to:

determine first key-based matches between the generated first lookup key with the first offset values and the generated second lookup key with the second offset values;

compute a first similarity score for a pair of the first set of image frames and the second set of image frames,

wherein the first similarity score is computed based on the determined first key-based matches; and

determine the match between the first set of image frames and the second set of image frames based on a determination that the computed first similarity score is above a threshold.

6. The system according to claim 1 , wherein the circuitry is further configured to:

extract, from the received set of sensor data, a first audio data corresponding to the first image sequence;

generate a first sonogram of the first audio data;

filter the generated first sonogram based on one of a second two-dimensional (2D) diamond kernel or a second masked max filter to generate a third filter result;

convert the third filter result to a third list of one of the time-domain values or the frequency-domain values; and

generate a third lookup key with third offset values to neighboring list elements of the third list.

7. The system according to claim 6 , wherein the circuitry is further configured to:

extract, from the received set of sensor data, a second audio data corresponding to the second image sequence;

generate a second sonogram of the second audio data;

filter the generated second sonogram based on one of the second 2D diamond kernel or the second masked max filter to generate a fourth filter result;

convert the fourth filter result to a fourth list of one of the time-domain values or the frequency-domain values; and

generate a fourth lookup key with fourth offset values to neighboring list elements of the fourth list.

8. The system according to claim 7 , wherein the circuitry is further configured to:

determine second key-based matches between the generated third lookup key with the third offset values and the generated fourth lookup key with the fourth offset values;

compute a second similarity score for a pair of the first set of image frames and the second set of image frames,

wherein the second similarity score is computed based on the determined second key-based matches; and

determine the match between the first set of image frames and the second set of image frames based on a determination that the computed second similarity score is above a threshold.

9. The system according to claim 1 , wherein the computed offset corresponds to a number of frames by which a positional identifier of a first image frame of the first set of image frames one of precedes or succeeds a positional identifier of a second frame of the second set of image frames.

10. The system according to claim 1 , wherein the synchronization comprises a shift in a positional identifier of the first set of image frames in the first image sequence with respect to a positional identifier of the second set of image frames in the second image sequence.

11. The system according to claim 1 , wherein the synchronization comprises a down-sampling of at least one of the first image sequence or the second image sequence, such that a positional identifier of the first set of image frames matches with a positional identifier of the second set of image frames.

12. A method, comprising:

receiving, from a plurality of image-capture devices, image data comprising a plurality of image sequences of a first object, wherein each image sequence of the plurality of image sequences corresponds to an image-capture device of the plurality of image-capture devices;

receiving a set of sensor data from the plurality of image-capture devices,

wherein each sensor data of the received set of sensor data comprises at least one of an Inertial Measurement Unit (IMU) data or audio data, and

each sensor data of the received set of sensor data is associated with a duration of acquisition of a corresponding image sequence of the plurality of image sequences;

extracting, from the received set of sensor data, a first IMU data corresponding to a first image sequence of the plurality of image sequences;

generating a first spectrogram of the first IMU data;

filtering the generated first spectrogram based on one of a first two-dimensional (2D) diamond kernel or a first masked max filter to generate a first filter result;

converting the first filter result to a first list of one of time-domain values or frequency-domain values;

generating a first lookup key with first offset values to neighboring list elements of the first list;

determining, based on the generated first lookup key, a match between a first set of image frames of the first image sequence and a second set of image frames of a second image sequence of the plurality of image sequences;

computing an offset between the first set of image frames and the second set of image frames, based on the match; and

synchronizing the first image sequence with the second image sequence based on the computed offset.

13. The method according to claim 12 , further comprising controlling the plurality of image-capture devices to acquire the plurality of image sequences from a corresponding plurality of viewpoints.

14. The method according to claim 12 , further comprising:

extracting, from the received set of sensor data, a second IMU data corresponding to the second image sequence;

generating a second spectrogram of the second IMU data;

filtering the generated second spectrogram based on one of the first 2D diamond kernel or the first masked max filter to generate a second filter result;

converting the second filter result to a second list of one of the time-domain values or the frequency-domain values; and

generating a second lookup key with second offset values to neighboring list elements of the second list.

15. The method according to claim 14 , further comprising:

determining first key-based matches between the first lookup key with the first offset values and the second lookup key with the second offset values;

computing a first similarity score for a pair of the first set of image frames and the second set of image frames, wherein the first similarity score is computed based on the determined first key-based matches; and

determining the match between the first set of image frames and the second set of image frames based on a determination that the computed first similarity score is above a threshold.

16. The method according to claim 12 , further comprising:

extracting, from the received set of sensor data, a first audio data corresponding to the first image sequence;

extracting, from the received set of sensor data, a second audio data corresponding to the second image sequence;

generating a first sonogram of the first audio data and a second sonogram of the second audio data;

filtering the first sonogram based on one of a second two-dimensional (2D) diamond kernel or a second masked max filter to generate a third filter result and the second sonogram based on one of the second 2D diamond kernel or the second masked max filter to generate a fourth filter result;

converting the third filter result to a third list of one of the time-domain values or the frequency-domain values and the fourth filter result to a fourth list of one of the time-domain values or the frequency-domain values; and

generating a third lookup key with third offset values to neighboring list elements of the third list and a fourth lookup key with fourth offset values to neighboring list elements of the fourth list.

17. The method according to claim 16 , further comprising:

determining second key-based matches between the third lookup key with the third offset values and the fourth lookup key with the fourth offset values;

computing a second similarity score for a pair of the first set of image frames and the second set of image frames, wherein the second similarity score is computed based on the determined second key-based matches; and

determining the match between the first set of image frames and the second set of image frames based on a determination that the computed second similarity score is above a threshold.

18. The method according to claim 12 , wherein the synchronization comprises a shift in a positional identifier of the first set of image frames in the first image sequence with respect to a positional identifier of the second set of image frames in the second image sequence.

19. A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a system, causes the system to execute operations, the operations comprising:

receiving, from a plurality of image-capture devices, image data comprising a plurality of image sequences of a first object, wherein each image sequence of the plurality of image sequences corresponds to an image-capture device of the plurality of image-capture devices;

receiving a set of sensor data from the plurality of image-capture devices,

wherein each sensor data of the received set of sensor data comprises at least one of an Inertial Measurement Unit (IMU) data or audio data, and

each sensor data of the received set of sensor data is associated with a duration of acquisition of a corresponding image sequence of the plurality of image sequences;

extracting, from the received set of sensor data, a first IMU data corresponding to a first image sequence of the plurality of image sequences;

generating a first spectrogram of the first IMU data;

filtering the generated first spectrogram based on one of a first two-dimensional (2D) diamond kernel or a first masked max filter to generate a first filter result;

converting the first filter result to a first list of one of time-domain values or frequency-domain values;

generating a first lookup key with first offset values to neighboring list elements of the first list;

determining, based on the generated first lookup key, a match between a first set of image frames of the first image sequence and a second set of image frames of a second image sequence of the plurality of image sequences;

computing an offset between the first set of image frames and the second set of image frames, based on the match; and

synchronizing the first image sequence with the second image sequence based on the computed offset.

20. A system, comprising:

circuitry configured to:

receive, from a plurality of image-capture devices, image data that comprises a plurality of image sequences of a first object, wherein each image sequence of the plurality of image sequences corresponds to an image-capture device of the plurality of image-capture devices;

receive a set of sensor data from the plurality of image-capture devices,

wherein each sensor data of the received set of sensor data comprises at least one of an Inertial Measurement Unit (IMU) data or audio data, and

each sensor data of the received set of sensor data is associated with a duration of acquisition of a corresponding image sequence of the plurality of image sequences;

extract, from the received set of sensor data, a first audio data corresponding to a first image sequence;

generate a sonogram of the first audio data;

filter the generated sonogram based on one of a two-dimensional (2D) diamond kernel or a masked max filter to generate a filter result;

convert the filter result to a list of one of time-domain values or frequency-domain values;

generate a lookup key with offset values to neighboring list elements of the list;

determine, based on the generated lookup key, a match between a first set of image frames of the first image sequence and a second set of image frames of a second image sequence of the plurality of image sequences;

compute an offset between the first set of image frames and the second set of image frames, based on the match; and

synchronize the first image sequence with the second image sequence based on the computed offset.