IP Library Granted Patent US 9,532,099
Granted Patent B2
US 9,532,099 · App. 14/667,525 · Granted Dec 27, 2016

Distributed media stream synchronization control

Inventors: Sean J. Lawrence (Bangalore, IN); Raghavendra Angadimani (Bangalore, IN)
Assignee: Intel Corporation
H04N21/4307H04N5/04H04N21/436
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,532,099
App. No.
14/667,525
Granted
Dec 27, 2016
Kind
B2
Abstract

One or more system, apparatus, method, and computer readable media is described below for wireless display synchronization of audio and video data streams received through a direct wireless link between a content source and sink. In some embodiments, the presentation time stamp (PTS) associated with the encoding of a digital audio and digital video data stream into a compressed packetized data stream at the source is utilized as a control point for synchronization of the digital audio and video stream payloads presented at the sink. In some embodiments, one or more wireless display-synchronized PTS values are determined based on a feedback signal indicative of display synchronization error. A source including an optical camera may generate audio/video (A/V) streams that encoded and packetized by a multiplexer with wireless display-synchronized PTS values. In further embodiments, a media sync user input interface is provided at one or more of the source and sink.

Claims (82)

1. A video source apparatus, comprising:

a stream multiplexer to assign a presentation timestamp (PTS) to one or more first packets of encoded video data, and to one or more first packets of encoded audio data;

a wireless transmitter coupled to an output of the multiplexer to wirelessly stream the first packets to a paired display device; and

a PTS adjustment module coupled to an input of the multiplexer to trigger a PTS modification for one or more second packets of the encoded video and audio data that is responsive to an audio/video (A/V) presentation synchronization error perceived or determined based on an A/V output of the display device rendered from the first packets.

2. The apparatus of claim 1 , wherein the PTS adjustment module is to:

receive a synchronization error signal indicative of the A/V presentation synchronization error; and

determine, based on the synchronization error signal, the PTS modification that counters the A/V presentation synchronization error.

3. The apparatus of claim 2 , wherein:

the PTS adjustment module is to:

map the synchronization error signal to a predetermined PTS offset value;

generate a modified PTS value by adding the predetermined PTS offset value to a reference PTS value; and

the multiplexer is to assign the modified PTS value to one or more of the second packets of the encoded video and audio data.

4. The apparatus of claim 2 , wherein:

the multiplexer is to assign a modified PTS including a PTS offset value to delay presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets.

5. The apparatus of claim 1 , further comprising:

a synchronization error user interface to receive a user input indicative of the A/V presentation synchronization error; and

wherein the PTS adjustment module is to:

map the user input to a predetermined PTS offset value;

generate a modified PTS value by adding the predetermined PTS offset value to a reference PTS value; and

wherein the multiplexer is to:

assign a modified PTS based on a PTS offset value that is to delay presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets; and

assign a modified PTS based on a PTS offset value that is to delay presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

6. The apparatus of claim 1 , further comprising:

a camera module to generate digital video and audio data;

a video encoder and audio encoder coupled between an output of the camera module and an input of the multiplexer, the encoders to generate the encoded video and audio data.

7. The apparatus of claim 1 , further comprising:

a video encoder and audio encoder coupled to an input of the multiplexer, the encoders to generate the encoded video and audio data;

an A/V decoder coupled to an input of the video and audio encoders, wherein the A/V decoder, video encoder, and audio encoder comprise a transcoder to receive a first encoded A/V stream and output second encoded audio and video bit streams.

8. A wireless display system, comprising:

the video source apparatus of claim 1 ; and

a display device to communicate the A/V presentation synchronization error over a back channel maintained between the A/V source device and the display device.

9. The wireless display system of claim 8 , wherein the display apparatus further comprises a synchronization error user interface to receive a user input indicative of the A/V presentation synchronization error.

10. A wireless video display, comprising:

a wireless receiver to receive a stream of first packets from a paired source device;

a de-multiplexer coupled to an output of the wireless receiver, and to de-multiplex a compressed audio data payload from a compressed video data payload, based at least in part, on one or more presentation timestamp (PTS) associated with the first packets;

an audio decoder and video decoder coupled to an output of the de-multiplexer, the decoders to generate digital audio and video data;

an audio rendering pipeline and video rendering pipeline coupled to the decoders to output an audio/visual (A/V) representation of the digital audio and video data;

a synchronization error user interface to receive a user input indicative of a synchronization error in the A/V presentation; and

a wireless transmitter to communicate an indication of the A/V presentation synchronization error to the paired source device.

11. The display of claim 10 , wherein the wireless transmitter is to send a signal indicative of the A/V presentation error over a back channel maintained with the paired source device.

12. A method for controlling synchronization of audio/video (A/V) presentation by a wireless display, the method comprising:

assigning a presentation timestamp (PTS) to one or more first packets of encoded video data, and to one or more first packets of encoded audio data;

wirelessly streaming the first packets to a paired display device;

assigning a display-synchronized PTS to one or more second packets of the encoded video and audio data that is responsive to an A/V presentation synchronization error perceived or determined based on an A/V output of the display device rendered from the first packets; and

wirelessly streaming the second packets to the paired display device.

13. The method of claim 12 , further comprising:

receiving a synchronization error signal indicative of the A/V presentation synchronization error; and

determining, based on the synchronization error signal, the display-synchronized PTS that counters the A/V presentation synchronization error.

14. The method of claim 13 , wherein determining the display-synchronized PTS further comprises:

mapping the synchronization error signal to a predetermined PTS offset value;

generating a modified PTS value by adding the predetermined PTS offset value to a reference PTS value; and

assigning the modified PTS value to one or more of the second packets of the encoded video and audio data associated with the reference PTS value.

15. The method of claim 12 , wherein assigning a display-synchronized PTS further comprises:

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets; and

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

16. The method of claim 12 , further comprising:

presenting a synchronization error user interface;

receiving, through the synchronization error user interface, a user input indicative of the A/V presentation synchronization error;

mapping the user input to a predetermined PTS offset value;

generating a modified PTS by adding the predetermined PTS offset value to a reference PTS; and

assigning the modified PTS to delay presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets, or to delay presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

17. The method of claim 12 , further comprising:

generating the digital video and digital audio data with a camera module including an image sensor;

and encoding the digital video and digital audio data into a compressed format.

18. The method of claim 12 , further comprising:

communicating the A/V presentation synchronization error over a back channel maintained between the A/V source device and the display device.

19. One or more non-transitory computer readable media including instruction stored thereon, which when executed by a processing system, cause the system to perform a method comprising:

assigning a presentation timestamp (PTS) to one or more first packets of encoded video data, and to one or more first packets of encoded audio data;

wirelessly streaming the first packets to a paired display device;

assigning a display-synchronized PTS to one or more second packets of the encoded video and audio data that is responsive to an audio/video (A/V) presentation synchronization error perceived or determined based on an A/V output of the display device rendered from the first packets; and

wirelessly streaming the second packets to the paired display device.

20. The media of claim 19 , further including instructions stored thereon, which when executed by the processing system, cause the system to perform a method comprising:

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets; and

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

21. A method for controlling synchronization of audio/video (A/V) presentation by a wireless display, the method comprising:

wirelessly receiving a stream of first packets from a paired source device;

de-multiplexing a compressed audio data payload from a compressed video data payload, based at least in part, on one or more presentation timestamp (PTS) associated the first packets;

decoding the compressed audio and video data payloads into digital audio and video data;

rendering an A/V representation of the digital audio and video data;

receiving a user input indicative of a synchronization error in the A/V presentation; and

wirelessly transmitting an indication of the A/V presentation synchronization error to the paired source device.

22. The method of claim 21 , wherein wirelessly transmitting the indication further comprises sending a signal indicative of the A/V presentation error over a back channel maintained with the paired source device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 13, 2016
From: LAWRENCE, SEAN J.; ANGADIMANI, RAGHAVENDRA
To: INTEL CORPORATION
Reel/Frame 037477/0296 →
Continuity (1)
Related Publication 20160286260A1 · Sep 29, 2016