IP Library Granted Patent US 8,994,792
Granted Patent B2
US 8,994,792 · App. 13/077,912 · Granted Mar 31, 2015

Method and system for creating a 3D video from a monoscopic 2D video and corresponding depth information

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 8,994,792
App. No.
13/077,912
Granted
Mar 31, 2015
Kind
B2
Abstract

Image sensors and a depth sensor of a monoscopic video sensing device are utilized to capture a 2D video and corresponding depth information. Regions of interest (ROIs) for the captured 2D video are selected based on the captured corresponding depth information. The monoscopic video sensing device selectively processes the captured 2D video and the captured corresponding depth information based on the selected ROIs. A 3D video is composed from the processed 2D video for display. The captured depth information that is synchronized to the captured 2D video is stored as metadata, and may be interpolated to match video resolution of the captured 2D video. The captured 2D video and the captured corresponding depth information are enhanced through scalable video coding. With 3D video rendering, relevant image/video components of the captured 2D video are selected based on the selected ROIs to compose the 3D video together with the corresponding depth information.

Claims (40)

1. A method, comprising:

capturing a video via one or more image sensors of a monoscopic video sensing device;

capturing corresponding depth information for said captured video via a depth sensor of said monoscopic video sensing device;

synchronizing said captured corresponding depth information to said captured video;

selecting one or more regions of interest in said captured video; and

composing a modified video from said captured video and said captured corresponding depth information, wherein said composing comprises scalable video coding, which compresses said captured video and said captured depth information into a base layer video and an enhancement layer video respectively wherein, depth information corresponding to said one or more regions of interest is allocated more bits for compression than rest of said depth information.

2. The method according to claim 1 , comprising storing said synchronized depth information as metadata for said captured video.

3. The method according to claim 2 , comprising matching resolution of said stored depth information to video resolution of said captured video.

4. The method according to claim 3 , comprising decompressing said compressed video and said compressed depth information; and determining relevance of video components within said decompressed video based on said one or more regions of interest.

5. The method according to claim 1 , further comprising mapping the captured video to the modified video based on depth information contained in the one or more regions of interest.

6. The method according to claim 1 , wherein said captured video is a two-dimensional video and the modified video is a three-dimensional video based on said captured two-dimensional video and said corresponding captured depth information.

7. A system for processing signals, the system comprising:

one or more processors and/or circuits for use in a monoscopic video sensing device, said one or more processors and/or circuits comprising one or more image sensors and a depth sensor, wherein said one or more processors and/or circuits are operable to:

capture a video via said one or more image sensors;

capture depth information for said captured video via said depth sensor;

synchronize said captured depth information to said captured video;

select one or more regions of interest in said captured video and/or said captured depth information;

identify video and depth information corresponding to said regions of interest;

process said captured video and said captured depth information selectively based on said one or more regions of interest, said process comprising encoding of said captured video and depth information, wherein said regions of interest are encoded at a lower compression ratio than rest of said captured video and depth information, and wherein, said captured video and said stored depth information are encoded into a base layer video and an enhancement layer video, respectively; and

compose an enhanced video from said processed video and said processed depth information.

8. The system according to claim 7 , wherein said one or more circuits are operable to store said synchronized depth information as metadata for said captured video.

9. The system according to claim 8 , wherein said one or more circuits are operable to match resolution of said stored depth information to video resolution of said captured video.

10. The system according to claim 8 , wherein said one or more circuits are operable to:

decompress said compressed video and said compressed depth information; and

determine relevance of video components within said decompressed video based on said one or more regions of interest.

11. The system according to claim 7 , wherein the one or more image sensors comprise a two-dimensional image sensor and the depth sensor comprises a one-dimensional depth sensor.

12. The system according to claim 7 , wherein the processing of said captured video and said captured depth information further comprises allocation of more processing resources to process captured video and depth information in said one or more regions of interest than to process rest of said captured video and depth information.

13. The system according to claim 7 , wherein said captured video is a two-dimensional video and the composed enhanced video is a three-dimensional video based on said processed two-dimensional video and said processed depth information.

14. A device for processing video signals, said device comprising:

one or more circuits for capturing video, said one or circuits comprising an image sensor and a depth sensor, wherein said one or more circuits are operable to:

capture depth information via said depth sensor;

capture a video via said image sensor;

synchronize said captured depth information and said captured video;

select a depth-based region of interest in said captured depth information;

determine a region of video corresponding to said depth-based region of interest; and

selectively encode said video based on said depth-based region of interest wherein said region of video corresponding to said depth-based region of interest is allocated more bits than rest of said video, wherein said captured video is encoded in a base video layer and said depth information is encoded in an enhancement video layer.

15. The device according to claim 14 , wherein said one or more circuits are operable to:

generate an enhanced video based on said captured video and said depth information;

map said selected depth-based region of interest in said captured video to said generated video; and

selectively encode said generated video based on said mapped depth-based regions of interest, wherein information of said depth-based region of interest is allocated more bits than a region of non-interest.

Assignments (7)
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENT NUMBER 9,385,856 TO 9,385,756 PREVIOUSLY RECORDED AT REEL: 47349 FRAME: 001. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER. Recorded Mar 22, 2019
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 051144/0648 →
CORRECTIVE ASSIGNMENT TO CORRECT THE EFFECTIVE DATE PREVIOUSLY RECORDED ON REEL 047229 FRAME 0408. ASSIGNOR(S) HEREBY CONFIRMS THE THE EFFECTIVE DATE IS 09/05/2018. Recorded Oct 29, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047349/0001 →
MERGER Recorded Oct 4, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047229/0408 →
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS Recorded Feb 3, 2017
From: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
To: BROADCOM CORPORATION
Reel/Frame 041712/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 1, 2017
From: BROADCOM CORPORATION
To: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
Reel/Frame 041706/0001 →
PATENT SECURITY AGREEMENT Recorded Feb 11, 2016
From: BROADCOM CORPORATION
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 037806/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 13, 2011
From: CHEN, XUEMIN; SESHADRI, NAMBI; KARAOGUZ, JEYHAN; BOROSS, CHRIS
To: BROADCOM CORPORATION
Reel/Frame 026434/0787 →