Synchronization and mixing of audio and video streams in network-based video conferencing call systems
In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.
1. A computer-implemented method for synchronizing an audio stream and a video stream, the method comprising:
playing the audio stream as a series of audio chunks according to time markers for the audio chunks;
based on a comparison of a time marker for a current frame of the video stream to a calculated time period for playing of a current audio chunk, determining if the current frame of the video stream should occur during the playing of the current audio chunk:
if the current frame should occur during the time period calculated for the current audio chunk, then displaying the current frame and moving to a next frame;
if the current frame should occur after the time period, then waiting; and
if the current frame should have occurred before the time period, then discarding the current frame and moving to a next frame,
wherein the step of determining if a current frame of the video stream should occur during the playing of the current audio chunk comprises:
determining a time marker for the current video frame;
determining a start time and an end time for the time period calculated for the audio chunk, the determining comprising:
determining a nominal start time and a nominal end time for the audio chunk,
adjusting the nominal start time by a tolerance, and
adjusting the nominal end time by another tolerance; and
determining whether the time marker falls between the start time and the end time.
2. A computer program product for synchronizing an audio stream and a video stream, wherein the computer program product is stored on a non-transitory computer-readable medium that includes instructions that, when loaded into memory, cause a processor to perform a method, the method comprising:
playing the audio stream as a series of audio chunks according to time markers for the audio chunks;
based on a comparison of a time marker for a current frame of the video stream to a calculated time period for playing of a current audio chunk, determining if the current frame of the video stream should occur during the playing of the current audio chunk:
if the current frame should occur during the time period calculated for the current audio chunk, then displaying the current frame and moving to a next frame;
if the current frame should occur after the time period, then waiting; and
if the current frame should have occurred before the time period, then discarding the current frame and moving to a next frame,
wherein the step of determining if a current frame of the video stream should occur during the playing of the current audio chunk comprises:
determining a time marker for the current video frame;
determining a start time and an end time for the time period calculated for the audio chunk, the determining comprising:
determining a nominal start time and a nominal end time for the audio chunk,
adjusting the nominal start time by a tolerance, and
adjusting the nominal end time by another tolerance; and
determining whether the time marker falls between the start time and the end time.
3. A non-transitory computer-readable medium that includes instructions that, when loaded into memory, cause a processor to perform a method for synchronizing an audio stream and a video stream, the method comprising:
playing the audio stream as a series of audio chunks according to time markers for the audio chunks;
based on a comparison of a time marker for a current frame of the video stream to a calculated time period for playing of a current audio chunk, determining if the current frame of the video stream should occur during the playing of the current audio chunk:
if the current frame should occur during the time period calculated for the current audio chunk, then displaying the current frame and moving to a next frame;
if the current frame should occur after the time period, then waiting; and
if the current frame should have occurred before the time period, then discarding the current frame and moving to a next frame,
wherein the step of determining if a current frame of the video stream should occur during the playing of the current audio chunk comprises:
determining a time marker for the current video frame;
determining a start time and an end time for the time period calculated for the audio chunk, the determining comprising:
determining a nominal start time and a nominal end time for the audio chunk,
adjusting the nominal start time by a tolerance, and
adjusting the nominal end time by another tolerance; and
determining whether the time marker falls between the start time and the end time.