IP Library Granted Patent US 9,013,552
Granted Patent B2
US 9,013,552 · App. 13/174,261 · Granted Apr 21, 2015

Method and system for utilizing image sensor pipeline (ISP) for scaling 3D images based on Z-depth information

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,013,552
App. No.
13/174,261
Granted
Apr 21, 2015
Kind
B2
Abstract

A monoscopic video camera may capture, via at least one image sensor, two-dimensional video, and may capture, via at least one depth sensor, corresponding depth information for the captured two-dimensional video. The monoscopic video camera may then adaptively configure scaling operations applicable to the captured two-dimensional video based on the depth information, which may comprise variably scaling different portions of the two-dimensional video. In this regard, the monoscopic video camera may determine, based on the depth information, a plurality of depth planes. The different portions of the two-dimensional video that are subjected to variable scaling may be determined based on the plurality of depth planes. Configuration of scaling operations may be performed in response to user input, which may comprise a zoom command. In this regard, scaling operations may be configured to focus on one or more of the different portions of the two-dimensional video based on zoom commands.

Claims (44)

1. A method, comprising:

capturing via at least one image sensor of a monoscopic video camera, a two-dimensional video;

capturing via at least one depth sensor of said monoscopic video camera, corresponding depth information for said captured two-dimensional video; and

focusing on a portion among a plurality of portions of said two-dimensional video, wherein focusing comprises selecting said portion as a focal point for processing said two-dimensional video; and

scaling said two-dimensional video in response to a user-zoom, wherein scaling comprises resizing said portions of said two-dimensional video variably by:

scaling said focused portion at a focal depth plane using a first scale factor, the first factor generated based on said user-zoom: and

scaling another portion among said plurality of portions at another depth plane using a second scale factor, the second scale factor generated to be proportional to said first scale factor based on a relation between said focal depth plane and said another depth plane.

2. The method according to claim 1 , wherein, said variable scaling comprises adjusting and/or controlling one or more functions in an image sensor pipeline (ISP) utilized during image processing in said monoscopic video camera.

3. The method according to claim 1 , comprising:

determining a plurality of depth planes in said depth information; and

determining said plurality of portions of said two-dimensional video that are subjected to variable scaling based on said determined plurality of depth planes.

4. The method according to claim 1 , wherein said user-zoom comprises a user-input zoom factor.

5. The method according to claim 1 , comprising synchronizing said captured corresponding depth information to said captured two-dimensional video.

6. The method according to claim 1 , comprising:

composing three-dimensional video from said captured two-dimensional video based on said captured corresponding depth information; and

rendering said composed three-dimensional video via a display that is communicatively coupled to said monoscopic video camera.

7. A system, comprising:

one or more circuits for use in a monoscopic video camera that comprises at least one image sensor and at least one depth sensor, said one or more circuits being operable to:

capture via said at least one image sensor of the monoscopic video camera, a two-dimensional video;

capture via said at least one depth sensor of said monoscopic video camera, corresponding depth information for said captured two-dimensional video;

select a focal portion of said captured two-dimensional video;

identify a focal depth of said selected focal portion in said captured depth information; and

scale said two-dimensional video adaptively in response to a user-input zoom-factor and based on said depth information, said adaptive scaling comprising:

scaling said focal portion of said two-dimensional video with a first scale-factor, wherein said first scale-factor is calculated based on said zoom-factor; and

scaling a second portion of said two-dimensional video with a second scale-factor, wherein said second scale-factor is calculated based on said zoom-factor and proportion of depth of said second portion to said focal depth.

8. The system according to claim 7 , wherein said one or more circuits comprise an image sensor pipeline (ISP), and said one or more circuits are operable to perform said adaptive scaling by adjusting and/or controlling one or more functions in said image sensor pipeline (ISP) utilized during image processing in said monoscopic video camera.

9. The system according to claim 7 , wherein said one or more circuits are operable to determine a plurality of depth planes corresponding to said two-dimensional video based on said depth information.

10. The system according to claim 7 , wherein in response to said user input, said one or more circuits are operable to select a third portion of said two-dimensional video as said focal plane.

11. The system according to claim 7 , wherein said one or more circuits are operable to synchronize said captured corresponding depth information to said captured two-dimensional video.

12. The system according to claim 7 , wherein said one or more circuits are operable to compose three-dimensional video from said captured two-dimensional video based on said captured corresponding depth information.

13. The system according to claim 12 , wherein said one or more circuits are operable to render said composed three-dimensional video via a display that is communicatively coupled to said monoscopic video camera.

14. A device comprising:

an image sensor configured to capture a two-dimensional image;

a depth sensor configured to capture depth information corresponding to the captured two-dimensional image;

a processor further configured to operate the image sensor to select a portion of the two-dimensional image as a focal portion, wherein light from the focal portion is captured optimally;

the processor further operable to determine a focal depth of the focal portion from the captured depth information;

a control configured to receive a zoom command;

the processor configured to calculate a zoom-factor in response to the zoom command;

the processor further configured to calculate, in response to the zoom command, a first scale-factor to be applied to portions of the two-dimensional image at the focal depth, the scale-factor based on the zoom-factor and the focal depth;

the processor further configured to scale the portions in the two-dimensional image at the focal depth using the calculated first scale-factor in response to the zoom command;

the processor further configured to calculate a second scale-factor for a second portion in the two-dimensional image at a different depth than the focal depth, the second scale-factor based on the zoom-factor and a proportion of the focal depth and the depth of the second portion; and

the processor further configured to scale the second portion in the captured two-dimensional image using the second scale-factor.

15. The device of claim 14 , wherein the processor is further configured to:

operate the the image sensor to select another portion of the two-dimensional image as the focal portion in response to the zoom.

Assignments (6)
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENT NUMBER 9,385,856 TO 9,385,756 PREVIOUSLY RECORDED AT REEL: 47349 FRAME: 001. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER. Recorded Mar 22, 2019
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 051144/0648 →
CORRECTIVE ASSIGNMENT TO CORRECT THE EFFECTIVE DATE PREVIOUSLY RECORDED ON REEL 047229 FRAME 0408. ASSIGNOR(S) HEREBY CONFIRMS THE THE EFFECTIVE DATE IS 09/05/2018. Recorded Oct 29, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047349/0001 →
MERGER Recorded Oct 4, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047229/0408 →
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS Recorded Feb 3, 2017
From: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
To: BROADCOM CORPORATION
Reel/Frame 041712/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 1, 2017
From: BROADCOM CORPORATION
To: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
Reel/Frame 041706/0001 →
PATENT SECURITY AGREEMENT Recorded Feb 11, 2016
From: BROADCOM CORPORATION
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 037806/0001 →