Method and system for precise synchronization of audio and video streams during a distributed communication session with multiple participants
View Patent ↗Described are the architecture of such a system, algorithms for time synchronization during a multiway conferencing session, methods to fight with network imperfections such as jitter to improve synchronization, methods of introducing buffering delays to create handicaps for players with faster connections, methods which help players with synchronization (such as a synchronized metronome during a music conferencing session), methods for synchronized recording and live delivery of synchronized data to the audience watching the distributed interaction live over the Internet.
1. A method for clock synchronization by calculating a correction value to be added or subtracted from a local time calculated comprising:
defining a first computer A's time as an absolute time T A and a second computer N's time as non-absolute time T N ;
assigning a current best approximation of a network round trip time between computers A and N to RTT app ;
setting initial values of registers t A,last and t N,last to zero for computer A;
exchanging synchronization packets periodically;
computer N computing t 1 =T LocalTimeN , wherein T LocalTimeN is a local time of computer N;
computer N writing t 1 into a synchronization section in a synchronization request;
computer N sending an audio packet to computer A;
computer A receiving the audio packet;
computer A computing t 2 =T LocalTimeA , wherein T LocalTimeA is a local time of computer A;
computer A setting t A,last to the value of t 2 and setting t N,last to the value of t 1 from the audio packet if t 1 >t N,last ;
computer A calculating t′ 2 and writing (t′ 1 =t 1 +t′ 2 −t 2 , t′ 2 ) into the synchronization section in a synchronization reply if the t N,last was not sent in a synchronization reply, wherein t′ 2 is a local time of computer N just before an audio packet is sent to computer N;
computer A sending an audio packet to computer N;
computer N receiving the audio packet;
computer N computing t 3 =T LocalTimeN ;
computer N computing RTT=t 3 −t′ 1 ;
computer N assigning RTT app =RTT if RTT app is not defined;
computer N assigning the value of minimum(RTT app +RTT drift , RTT) to RTT app if RTT<1.05*RTT app , wherein RTT drift is a drift in packet round trip time;
assigning a mean value of (t′ 2 −(t′ 1 +t 3 )/2) of buffered data from previous synchronization cycles to ΔT N−A ;
synchronizing clock by adding a correction value ΔT N−A to a local time T N .
2. The method of claim 1 , wherein the local time T N is calculated by processes including:
using a plurality of samples of digitized audio stream which comes from a network at any given time as a primary tool for synchronizing audio streams, wherein a length of incoming sound is divided by an audio driver into a plurality of buffers of a constant time duration T Buffersize and an audio time increases with steps equal to T Buffersize ;
setting T Start equal to T SystemTime , wherein T SystemTime is a current time on a local computer;
setting T Last Local Time Result to zero;
calculating T Local time =maximum(T Last Local Time Result , minimum(T SystemTime −T Start , T Buffersize )+N Samples /R Sample ), wherein N Samples is a number of audio samples and R Sample is a sample rate;
storing T Local Time to T N .
3. The method of claim 1 , wherein the period of synchronization cycle ranges from 3 to 12 seconds.
4. The method of claim 1 , wherein the mean value of (t′ 2 −(t′ 1 +t 3 )/2) of buffered data from previous synchronization cycles is calculated using 1000 or less data samples.
5. The method of claim 1 , further comprising discarding a synchronization cycle if RTT>D=1.05*RTT app .
6. The method of claim 1 , further comprising:
assigning a value obtained by 1 ms divided by the synchronization cycle rate in kHz to RTT drift .
7. The method of claim 1 , further comprising:
assigning ΔT N =ΔT N−A /2;
calculating ΔT A−N treating computer N as having an absolute time;
assigning ΔT A =ΔT A−N /2.
8. The method of claim 7 , wherein both computer A and N make synchronization replies only after receiving synchronization requests.
9. The method of claim 7 further comprising:
calculating ΔTi=ΣΔT i−j /M, wherein ΔT i is a correction value for a computer i for synchronization among M computers, wherein M is an integer greater than 2 and ΔT i−i is zero.
10. The method of claim 9 further comprising sending synchronization replies to other computers in a round-robin manner.
11. The method of claim 9 further comprising sending an identification number of a sender of the synchronization request for the synchronization reply.
12. A system comprising:
a plurality of computers used for a distributed communication session with multiple participants using audio and video streams, wherein each computer includes:
a processor;
a clock;
a computer readable medium with instructions to carry out processes including:
defining a first computer A's time as an absolute time T A and a second computer N's time as non-absolute time T N ;
assigning a current best approximation of a network round trip time between computers A and N to RTT app ;
setting initial values of registers t A,last and t N,last to zero for computer A;
exchanging synchronization packets periodically;
computer N computing t 1 =T LocalTimeN , wherein T LocalTimeN is a local time of computer N;
computer N writing t 1 into a synchronization section in a synchronization request;
computer N sending an audio packet to computer A;
computer A receiving the audio packet;
computer A computing t 2 =T LocalTimeA , wherein T LocalTimeA is a local time of computer A;
computer A setting t A,last to the value of t 2 and setting t N,last to the value of t 1 from the audio packet if t 1 >t N,last ;
computer A calculating t′ 2 and writing (t′ 1 =t 1 +t′ 2 −t 2 , t′ 2 ) into the synchronization section in a synchronization reply if the t N,last was not sent in a synchronization reply, wherein t′ 2 is a local time of computer N just before an audio packet is sent to computer N;
computer A sending an audio packet to computer N;
computer N receiving the audio packet;
computer N computing t 3 =T LocalTimeN ;
computer N computing RTT=t 3 −t′ 1 ;
computer N assigning RTT app =RTT if RTT app is not defined;
computer N assigning the value of minimum(RTT app +RTT drift , RTT) to RTT app if RTT<1.05*RTT app , wherein RTT drift is a drift in packet round trip time;
assigning a mean value of (t′ 2 −(t′ 1 +t 3 )/2) of buffered data from previous synchronization cycles to ΔT N−A ;
synchronizing clock by adding a correction value ΔT N−A to a local time T N ;
a network interface for communication among the plurality of computers.