IP Library Granted Patent US 9,351,028
Granted Patent B2
US 9,351,028 · App. 13/182,645 · Granted May 24, 2016

Wireless 3D streaming server

Inventors: Soham V. Sheth (San Diego, CA); Vijayalakshmi R. Raveendran (San Diego, CA); Judit Martinez Bauza (San Diego, CA); PhaniKumar K. Bhamidipati (San Diego, CA); Xun Luo (San Diego, CA)
Assignee: QUALCOMM Incorporated
H04N21/4122H04N13/0029H04N13/0059H04N13/0066H04N21/2381H04N21/4358H04N21/83H04N2013/0081H04N2213/003
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,351,028
App. No.
13/182,645
Granted
May 24, 2016
Kind
B2
Abstract

An apparatus, system, method, and computer program product for streaming 3D content from a wireless device to a remote 3D display for the viewing of the 3D content on a larger screen. In some aspects, a wireless streaming server may encode 3D motion picture content in a certain format, where each image frame includes a 2D view concatenated side-by-side with a complementary frame of depth information. The combination of the 2D view and the depth information are capable of being processed by a client display to generate a stereoscopic image representing a 3D view.

Claims (120)

1. A method of streaming media content from a streaming server, comprising:

providing three-dimensional (3D) media content at the streaming server, wherein the 3D media content at least includes a two-dimensional (2D) image and a depth map corresponding to the 2D image;

establishing a streaming session between the streaming server and a streaming client;

compressing the 2D image into a first stream using first encoding options;

compressing the depth map into a second stream using second encoding options different from the first encoding options; and

transmitting the compressed 2D image in the first stream and the compressed depth map in the second stream to the streaming client over the streaming session, for remote rendering and display of a 3D image,

wherein the streaming session comprises a real time streaming protocol (RTSP) session, and wherein the compressed 2D image in the first stream and the compressed depth map in the second stream are streamed on different real-time transport protocol (RTP) ports of the same RTSP session.

2. The method of claim 1 , further comprising rendering the media content for local playback at the streaming server.

3. The method of claim 2 , wherein the rendering comprises rendering the 2D image based on the 3D media content.

4. The method of claim 1 , further comprising extracting information about the media content, adapted to indicate that the media content comprises 3D content.

5. The method of claim 4 , wherein the information comprises meta-data associated with the media content.

6. The method of claim 4 , further comprising transmitting the information about the media content to the streaming client over the streaming session.

7. The method of claim 6 , wherein the transmitting of the information about the media content comprises providing an information element in a session description protocol (SDP) description of the media content.

8. The method of claim 1 , further comprising synthesizing the depth map from the 2D image frame.

9. The method of claim 1 , further comprising:

synthesizing the depth map from a pair of 2D images representing a stereoscopic image; and

utilizing one image of the pair of 2D images as the 2D image.

10. A streaming server, comprising:

a file parser configured to extract 3D media content from a content source, wherein the 3D media content at least includes a two-dimensional (2D) image and a depth map corresponding to the 2D image;

a multimedia messaging services module configured to establish a streaming session with a streaming client;

a depth estimator configured to compress the 2D image into a first stream using first encoding options and to compress the depth map into a second stream using second encoding options different from the first encoding options; and

a wireless transmitter configured to stream the compressed 2D image in a first stream and the compressed depth map in a second stream to the streaming client over the streaming session,

wherein the multimedia messaging services module is configured to establish a real time streaming protocol (RTSP) session with the streaming client, and wherein the wireless transmitter is configured to stream the compressed 2D image in the first stream and the compressed depth map in the second stream on different real-time transport protocol (RTP) ports of the same RTSP session.

11. The streaming server of claim 10 , further comprising a memory, wherein the content source comprises the memory.

12. The streaming server of claim 10 , wherein the file parser is further configured to extract meta-data corresponding to the 3D media content from the content source.

13. The streaming server of claim 12 , wherein the multimedia messaging services module is further configured to send the meta-data to the streaming client over the streaming session.

14. The streaming server of claim 10 , further comprising:

a display device; and

a tap-out module configured to branch the 3D media content to the display device.

15. A method of wireless communication, comprising:

receiving media content at a streaming client from a streaming server over a wireless streaming session, wherein the media content comprises a two-dimensional (2D) image stream and a separate corresponding depth map stream, wherein a 2D image and a corresponding depth map are separately and respectively compressed into the 2D image stream and depth map stream using different encoding options,

wherein the wireless streaming session comprises a real time streaming protocol (RTSP) session, and wherein the compressed 2D image stream and the compressed depth map stream are received on different real-time transport protocol (RTP) ports of the same RTSP session;

synthesizing a three-dimensional (3D) image based on the 2D image stream and the depth map stream; and

rendering the 3D image on a display.

16. The method of claim 15 , further comprising receiving information about the media content from the streaming server.

17. The method of claim 16 , wherein the information comprises meta-data adapted to indicate that the media content comprises 3D content.

18. The method of claim 15 , wherein the synthesizing comprises utilizing depth information based rendering (DIBR) to produce a stereoscopic image representing the 3D image.

19. A streaming client, comprising:

a wireless receiver configured to receive three-dimensional (3D) media content from a streaming server, wherein the 3D media content comprises a two-dimensional (2D) image stream and a separate corresponding depth map stream, wherein a 2D image and a corresponding depth map are separately and respectively compressed into the 2D image stream and the depth map stream using different encoding options, wherein the wireless streaming session comprises a real time streaming protocol (RTSP) session, and wherein the wireless receiver is configured to receive the compressed 2D image stream and the compressed depth map stream on different real-time transport protocol (RTP) ports of the same RTSP session;

a depth image based rendering (DIBR) module for generating a stereoscopic image that includes a left view and a right view based on the 2D image stream and the corresponding depth map stream; and

a display driver for rendering the stereoscopic image on a display.

20. An apparatus for streaming media content from a streaming server, comprising:

means for providing three-dimensional (3D) media content at the streaming server, wherein the 3D media content at least includes a two-dimensional (2D) image and a depth map corresponding to the 2D image;

means for establishing a streaming session between the streaming server and a streaming client;

means for compressing the 2D image into a first stream using first encoding options and for compressing the depth map into a second stream using second encoding options different from the first encoding options; and

means for transmitting the compressed 2D image in the first stream and the compressed depth map in the second stream to the streaming client over the streaming session, for remote rendering and display of a 3D image,

wherein the streaming session comprises a real time streaming protocol (RTSP) session, and wherein the compressed 2D image in the first stream and the compressed depth map in the second stream are streamed on different real-time transport protocol (RTP) ports of the same RTSP session.

21. The apparatus of claim 20 , further comprising means for rendering the media content for local playback at the streaming server.

22. The apparatus of claim 21 , wherein the means for rendering comprises means for rendering the 2D image based on the 3D media content.

23. The apparatus of claim 20 , further comprising means for extracting information about the media content, adapted to indicate that the media content comprises 3D content.

24. The apparatus of claim 23 , wherein the information comprises meta-data associated with the media content.

25. The apparatus of claim 23 , further comprising means for transmitting the information about the media content to the streaming client over the streaming session.

26. The apparatus of claim 25 , wherein the means for transmitting the information about the media content comprises means for providing an information element in a session description protocol (SDP) description of the media content.

27. The apparatus of claim 20 , further comprising means for synthesizing the depth map from the 2D image frame.

28. The apparatus of claim 20 , further comprising:

means for synthesizing the depth map from a pair of 2D images representing a stereoscopic image; and

means for utilizing one of the pair of 2D images as the 2D image frame.

29. An apparatus for wireless communication, comprising:

means for receiving media content at a streaming client from a streaming server over a wireless streaming session, wherein the media content comprises a two-dimensional (2D) image stream and a separate corresponding depth map stream, wherein a 2D image and a corresponding depth map are separately and respectively compressed into the 2D image stream and the depth map stream using different encoding options, wherein the wireless streaming session comprises a real time streaming protocol (RTSP) session, and wherein the means for receiving is for receiving the compressed 2D image stream and the compressed depth map stream on different real-time transport protocol (RTP) ports of the same RTSP session;

means for synthesizing a three-dimensional (3D) image based on the 2D image stream and the depth map stream; and

means for rendering the 3D image on a display.

30. The apparatus of claim 29 , further comprising means for receiving information about the media content from the streaming server.

31. The apparatus of claim 30 , wherein the information comprises meta-data adapted to indicate that the media content comprises 3D content.

32. The apparatus of claim 29 , wherein the means for synthesizing comprises means for utilizing depth information based rendering (DIBR) to produce a stereoscopic image representing the 3D image.

33. A non-transitory computer-readable medium storing computer executable code comprising:

code for providing three-dimensional (3D) media content at a streaming server, wherein the 3D media content at least includes a two-dimensional (2D) image and a depth map corresponding to the 2D image;

code for establishing a streaming session between the streaming server and a streaming client;

code for compressing the 2D image into a first stream using first encoding options;

code for compressing the depth map into a second stream using second encoding options different from the first encoding options; and

code for transmitting the compressed 2D image in the first stream and the compressed depth map in the second stream to the streaming client over the streaming session, for remote rendering and display of a 3D image,

wherein the streaming session comprises a real time streaming protocol (RTSP) session, and wherein the compressed 2D image in the first stream and the compressed depth map in the second stream are streamed on different real-time transport protocol (RTP) ports of the same RTSP session.

34. The computer-readable medium of claim 33 , further comprising code for rendering the media content for local playback at the streaming server.

35. The computer-readable medium of claim 34 , wherein the code for rendering comprises code for rendering the 2D image based on the 3D media content.

36. The computer-readable medium of claim 33 , further comprising code for extracting information about the media content, adapted to indicate that the media content comprises 3D content.

37. The computer-readable medium of claim 36 , wherein the information comprises meta-data associated with the media content.

38. The computer-readable medium of claim 36 , further comprising code for transmitting the information about the media content to the streaming client over the streaming session.

39. The computer-readable medium of claim 38 , wherein the code for transmitting the information about the media content comprises code for providing an information element in a session description protocol (SDP) description of the media content.

40. The computer-readable medium of claim 33 , further comprising code for synthesizing the depth map from the 2D image.

41. The computer-readable medium of claim 33 , further comprising:

code for synthesizing the depth map from a pair of 2D images representing a stereoscopic image; and

code for utilizing one of the pair of 2D images as the 2D image.

42. A non-transitory computer-readable medium storing computer executable code, comprising:

code for receiving media content at a streaming client from a streaming server over a wireless streaming session, wherein the media content comprises a two-dimensional (2D) image stream and a corresponding depth map stream, wherein a 2D image and a corresponding depth map are separately and respectively compressed into the 2D image stream and the depth map stream using different encoding options, wherein the wireless streaming session comprises a real time streaming protocol (RTSP) session, and wherein the code for receiving includes code for receiving the compressed 2D image stream and the compressed depth map stream on different real-time transport protocol (RTP) ports of the same RTSP session;

code for synthesizing a three-dimensional (3D) image based on the 2D image stream and the depth map stream; and

code for rendering the 3D image on a display.

43. The computer-readable medium of claim 42 , further comprising code for receiving information about the media content from the streaming server.

44. The computer-readable medium of claim 43 , wherein the information comprises meta-data adapted to indicate that the media content comprises 3D content.

45. The computer-readable medium of claim 42 , wherein the code for synthesizing comprises code for utilizing depth information based rendering (DIBR) to produce a stereoscopic image representing the 3D image.

46. An apparatus for streaming media content from a streaming server, comprising:

a processor and a memory coupled to the processor, wherein the processor is configured to:

provide three-dimensional (3D) media content at the streaming server, wherein the 3D media content at least includes a two-dimensional (2D) image and a depth map corresponding to the 2D image;

establish a streaming session between the streaming server and a streaming client;

compress the 2D image into a first stream using first encoding options;

compress the depth map into a second stream using second encoding options different from the first encoding options; and

transmit the compressed 2D image in the first stream and the compressed depth map in the second stream to the streaming client over the streaming session, for remote rendering and display of a 3D image,

wherein the streaming session comprises a real time streaming protocol (RTSP) session, and wherein the compressed 2D image in the first stream and the compressed depth map in the second stream are streamed on different real-time transport protocol (RTP) ports of the same RTSP session.

47. The apparatus of claim 46 , wherein the processing system is further configured to render the media content for local playback at the streaming server.

48. The apparatus of claim 47 , wherein the rendering comprises rendering the 2D image based on the 3D media content.

49. The apparatus of claim 46 , wherein the processing system is further configured to extract information about the media content, adapted to indicate that the media content comprises 3D content.

50. The apparatus of claim 49 , wherein the information comprises meta-data associated with the media content.

51. The apparatus of claim 49 , wherein the processing system is further configured to transmit the information about the media content to the streaming client over the streaming session.

52. The apparatus of claim 51 , wherein the transmitting of the information about the media content comprises providing an information element in a session description protocol (SDP) description of the media content.

53. The apparatus of claim 46 , wherein the processing system is further configured to synthesize the depth map from the 2D image.

54. The apparatus of claim 46 , wherein the processing system is further configured to

synthesize the depth map from a pair of 2D images representing a stereoscopic image; and

utilize one of the pair of 2D images as the 2D image.

55. An apparatus for wireless communication, comprising:

a processor and a memory coupled to the processor, wherein the processor is configured to:

receive media content at a streaming client from a streaming server over a wireless streaming session, wherein the media content comprises a two-dimensional (2D) image stream and a separate corresponding depth map stream, wherein a 2D image and a corresponding depth map are separately and respectively compressed into the 2D image stream and the depth map stream using different encoding options, wherein the wireless streaming session comprises a real time streaming protocol (RTSP) session, and wherein the code for receiving includes code for receiving the compressed 2D image stream and the compressed depth map stream on different real-time transport protocol (RTP) ports of the same RTSP session;

synthesize a three-dimensional (3D) image based on the 2D image stream and the depth map stream; and

render the 3D image on a display.

56. The apparatus of claim 55 , wherein the wireless streaming session comprises a real time streaming protocol (RTSP) session and the compressed 2D image stream and the compressed depth map stream are received on different real-time transport protocol (RTP) ports of the same RTSP session.

57. The apparatus of claim 55 , wherein the processor is further configured to receive information about the media content from the streaming server.

58. The apparatus of claim 57 , wherein the information comprises meta-data adapted to indicate that the media content comprises 3D content.

59. The apparatus of claim 55 , wherein the synthesizing comprises utilizing depth information based rendering (DIBR) to produce a stereoscopic image representing the 3D image.

60. The method of claim 1 , wherein the streaming server is a mobile device.

61. The method of claim 1 , further comprising:

extracting a stereoscopic image that includes a left view and a right view from the 3D media content; and

generating the 2D image and the depth map based on the stereoscopic image.

62. The method of claim 1 , wherein compressing the depth map into a second stream using second encoding options different from the first encoding options comprises increasing the QP value for the chroma component of the depth map, and inserting an Intra (I) frame at regular intervals utilizing H.264 encoding.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 14, 2011
From: SHETH, SOHAM V.; RAVEENDRAN, VIJAYALAKSHMI R.; MARTINEZ BAUZA, JUDIT; BHAMIDIPATI, PHANIKUMAR K.; LUO, XUN
To: QUALCOMM INCORPORATED
Reel/Frame 026593/0525 →
Continuity (1)
Related Publication 20130019024A1 · Jan 17, 2013