Distributed measurement of latency and synchronization delay between audio/video streams

1. A method for identifying real-time latency of audio/video streams, comprising:

buffering signatures of a reference audio/video stream and signatures of a test audio/video stream;

constructing a needle as a vector including a set of signatures of the reference audio/video stream, the set of signatures including a set of video signatures of the reference audio/video stream and one or more sets of audio signatures of the reference audio/video stream, wherein the needle is selected to include a most recent set of n signatures in the reference audio/video stream;

computing correlations of the needle to successive vectors of sets of signatures of the test audio/video stream using a correlation function that calculates relatedness of the needle vector to each of the successive vectors of the test audio/video stream, the correlation function comparing the set of video signatures of the reference audio/video stream to video signatures of the test audio/video stream and comparing the one or more sets of audio signatures to audio signatures of the test audio/video stream, wherein the successive sets of signatures of the test audio/video stream each include n signatures and the correlation function calculates how closely the needle of the reference audio/video stream and the vectors of the test audio/video stream are related;

identifying a synchronization offset between the test stream and the reference stream according to a maximum correlation point of the correlations of the needle to the successive sets of signatures of the test audio/video stream; and

aligning the reference audio/video stream and the test audio/video stream according to the synchronization offset.

2. The method of claim 1 , further comprising simultaneously displaying the reference audio/video stream and the test audio/video stream as aligned.

3. The method of claim 1 , wherein the video signatures are calculated according to an amount of activity within a frame and adjacent frames and the audio signatures are calculated by tracking a difference in audio samples over time.

4. The method of claim 1 , wherein the reference audio/video stream is calculated from an audio/video stream at a first point along a multimedia delivery chain, and the test audio/video stream is calculated from the audio/video stream at a second point along the multimedia delivery chain.

5. The method of claim 1 , wherein the needle is selected either to accommodate searching for different synchronization offset ranges or based on other statistical features.

6. The method of claim 1 , wherein the correlation function utilizes one or more of Pearson Linear Correlation Coefficient (PLCC), mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), peak signal-to-noise ratio (PSNR), Spearman's rank correlation coeffect (SRCC), or Kendall's rank correlation coefficient (KRCC).

7. The method of claim 1 , further comprising recomputing the synchronization offset responsive to the synchronization offset failing to provide a correlation coefficient above a predefined threshold correlation.

8. The method of claim 1 , further comprising preprocessing the audio/video streams into a common audio format and a common video format to assist in generating the signatures of the reference audio/video stream and the signatures of the test audio/video stream.

9. The method of claim 1 , further comprising:

determining an outlier metric for the synchronization offset in comparison to one or more previous synchronization offsets; and

updating the synchronization offset based on the outlier metric indicating that the synchronization offset is not an outlier.

10. The method of claim 9 , wherein the outlier metric includes determining a median absolute difference for the one or more previous synchronization offsets; and further comprising accepting the synchronization offset responsive to a difference between the synchronization offset and the median absolute difference being within a threshold value.

11. The method of claim 9 , wherein the outlier metric includes determining a mean absolute error of the synchronization offset compared to a mean of the one or more previous synchronization offsets; and further comprising accepting the synchronization offset responsive to the mean absolute error being less than two standard deviations from the mean.

12. A system for identifying real-time latency of audio/video streams, comprising:

a computing device programmed to:

buffer signatures of a reference audio/video stream and signatures of a test audio/video stream;

construct a needle as a vector including a set of signatures of the reference audio/video stream, the set of signatures including a set of video signatures of the reference audio/video stream and one or more sets of audio signatures of the reference audio/video stream, wherein the needle is selected to include a most recent set of n signatures in the reference audio/video stream;

compute correlations of the needle to successive vectors of sets of signatures of the test audio/video stream using a correlation function that calculates relatedness of the needle vector to each of the successive vectors of the test audio/video stream, the correlation function comparing the set of video signatures of the reference audio/video stream to video signatures of the test audio/video stream and comparing the one or more sets of audio signatures to audio signatures of the test audio/video stream, wherein the successive sets of signatures of the test audio/video stream each include n signatures and the correlation function calculates how closely the needle of the reference audio/video stream and the vectors of the test audio/video stream are related;

identify a synchronization offset between the test stream and the reference stream according to a maximum correlation point of the correlations of the needle to the successive sets of signatures of the test audio/video stream; and

align the reference audio/video stream and the test audio/video stream according to the synchronization offset.

13. The system of claim 12 , wherein the computing device is further programmed to simultaneously display the reference audio/video stream and the test audio/video stream as aligned.

14. The system of claim 12 , wherein the video signatures are each calculated according to an amount of activity within a frame and adjacent frames and the audio signatures are calculated by tracking a difference in audio samples over time.

15. The system of claim 12 , wherein the computing device is further programmed to:

calculate the reference audio/video stream from an audio/video stream at a first point along a multimedia delivery chain; and

calculate the test audio/video stream from the audio/video stream at a second point along the multimedia delivery chain.

16. The system of claim 12 , wherein the computing device is further programmed to select the needle either to accommodate searching for different synchronization offset ranges or based on other statistical features.

17. The system of claim 12 , wherein the correlation function utilizes one or more of Pearson Linear Correlation Coefficient (PLCC), mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), peak signal-to-noise ratio (PSNR), Spearman's rank correlation coeffect (SRCC), or Kendall's rank correlation coefficient (KRCC).

18. The system of claim 12 , wherein the computing device is further programmed to recompute the synchronization offset responsive to the synchronization offset failing to provide a correlation coefficient above a predefined threshold correlation.

19. The system of claim 12 , wherein the computing device is further programmed to preprocess the audio/video streams into a common audio format and a common video format to assist in generating the signatures of the reference audio/video stream and the signatures of the test audio/video stream.

20. The system of claim 12 , wherein the computing device is further programmed:

determine an outlier metric for the synchronization offset in comparison to one or more previous synchronization offsets; and

update the synchronization offset based on the outlier metric indicating that the synchronization offset is not an outlier.

21. The system of claim 20 , wherein the outlier metric includes determining a median absolute difference for the one or more previous synchronization offsets; and the computing device is further programmed to accept the synchronization offset responsive to a difference between the synchronization offset and the median absolute difference being within a threshold value.

22. The system of claim 20 , wherein the outlier metric includes determining a mean absolute error of the synchronization offset compared to a mean of the one or more previous synchronization offsets; and the computing device is further programmed to accept the synchronization offset responsive to the mean absolute error being less than two standard deviations from the mean.

23. A non-transitory computer-readable medium comprising instructions for identifying real-time latency of audio/video streams, that when executed by a processor of a computing device, cause the computing device to:

buffer signatures of a reference audio/video stream and signatures of a test audio/video stream;

construct a needle as a vector including a set of signatures of the reference audio/video stream, the set of signatures including a set of video signatures of the reference audio/video stream and one or more sets of audio signatures of the reference audio/video stream, wherein the needle is selected to include a most recent set of n signatures in the reference audio/video stream;

compute correlations of the needle to successive vectors of sets of signatures of the test audio/video stream using a correlation function that calculates relatedness of the needle vector to each of the successive vectors of the test audio/video stream, the correlation function comparing the set of video signatures of the reference audio/video stream to video signatures of the test audio/video stream and comparing the one or more sets of audio signatures to audio signatures of the test audio/video stream, wherein the successive sets of signatures of the test audio/video stream each include n signatures and the correlation function calculates how closely the needle of the reference audio/video stream and the vectors of the test audio/video stream are related;

identify a synchronization offset between the test stream and the reference stream according to a maximum correlation point of the correlations of the needle to the successive sets of signatures of the test audio/video stream; and

align the reference audio/video stream and the test audio/video stream according to the synchronization offset.

24. The medium of claim 23 , wherein the signatures include video signatures each calculated according to an amount of activity within a frame and adjacent frames and audio signatures calculated by tracking a difference in audio samples over time.

25. The medium of claim 23 , further comprising instructions that when executed by the processor of the computing device, cause the computing device to:

calculate the reference audio/video stream from an audio/video stream at a first point along a multimedia delivery chain; and

calculate the test audio/video stream from the audio/video stream at a second point along the multimedia delivery chain.

26. The medium of claim 23 , further comprising instructions that when executed by the processor of the computing device, cause the computing device to select the needle either to accommodate searching for different synchronization offset ranges or based on other statistical features.

27. The medium of claim 23 , wherein the correlation function utilizes one or more of Pearson Linear Correlation Coefficient (PLCC), mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), peak signal-to-noise ratio (PSNR), Spearman's rank correlation coeffect (SRCC), or Kendall's rank correlation coefficient (KRCC).

28. The medium of claim 23 , further comprising instructions that when executed by the processor of the computing device, cause the computing device to recompute the synchronization offset responsive to the synchronization offset failing to provide a correlation coefficient above a predefined threshold correlation.

29. The medium of claim 23 , further comprising instructions that when executed by the processor of the computing device, cause the computing device to preprocess the audio/video streams into a common audio format and a common video format to assist in generating the signatures of the reference audio/video stream and the signatures of the test audio/video stream.

30. The medium of claim 23 , further comprising instructions that when executed by the processor of the computing device, cause the computing device to:

determine an outlier metric for the synchronization offset in comparison to one or more previous synchronization offsets; and

update the synchronization offset based on the outlier metric indicating that the synchronization offset is not an outlier.

31. The medium of claim 30 , wherein the outlier metric includes determining a median absolute difference for the one or more previous synchronization offsets; and further comprising instructions that when executed by the processor of the computing device, cause the computing device to accept the synchronization offset responsive to a difference between the synchronization offset and the median absolute difference being within a threshold value.

32. The medium of claim 30 , wherein the outlier metric includes determining a mean absolute error of the synchronization offset compared to a mean of the one or more previous synchronization offsets; and further comprising instructions that when executed by the processor of the computing device, cause the computing device to accept the synchronization offset responsive to the mean absolute error being less than two standard deviations from the mean.

33. The method of claim 8 , wherein the common audio format is a mono signal at a common sample rate, and the common video format is a common resolution and common framerate.