IP Library Granted Patent US 8,446,516
Granted Patent B2
US 8,446,516 · App. 13/563,513 · Granted May 21, 2013

Generating and outputting video data from refocusable light field video data

Inventors: Colvin Pitts (Snohomish, WA); Yi-Ren Ng (Redwood City, CA)
Assignee: Lytro, Inc.
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 8,446,516
App. No.
13/563,513
Granted
May 21, 2013
Kind
B2
Abstract

Systems and methods are directed to acquiring, generating, manipulating and/or editing refocusable video data/frames. The refocusable video frames may be light field video frames that may be focused and/or refocused after acquisition or recording of such video frames. In one aspect, a method comprises: selecting a first key frame, wherein the first key frame corresponds to one of a plurality of refocusable light field video frames; selecting a second key frame, wherein the second key frame corresponds to one of the plurality of refocusable light field video frames which is temporally spaced apart from the first key frame such that a plurality of refocusable light field video frames are temporally disposed between the first and the second key frames; determining virtual focus parameters for the first key frame and the second key frame; and generating video data.

Claims (47)

1. A method of generating and outputting video data corresponding to a plurality of video data frames, the method comprising:

receiving refocusable light field video data, wherein the refocusable light field video data comprises a plurality of temporally contiguous refocusable light field video frames, wherein each refocusable light field video frame comprises an optical focus;

selecting a first key frame, wherein the first key frame corresponds to one of the plurality of refocusable light field video frames;

determining a virtual focus depth for the first key frame, wherein the virtual focus depth for the first key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the first key frame;

selecting a second key frame, wherein the second key frame corresponds to one of the plurality of refocusable light field video frames which is temporally spaced apart from the first key frame such that a plurality of refocusable light field video frames are temporally disposed between the first and the second key frames;

determining a virtual focus depth for the second key frame, wherein the virtual focus depth for the second key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the second key frame;

generating first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus depth for the first key frame, (ii) the virtual focus depth for the second key frame, (iii) an inter-frame interpolation, and (iv) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data comprises a plurality of video frames, each video frame comprising a virtual focus depth which is based on the virtual focus depth for the first key frame, the virtual focus depth for the second key frame, and inter-frame interpolation; and

outputting the first video data.

2. The method of claim 1 , wherein the inter-frame interpolation comprises linear interpolation.

3. The method of claim 1 , wherein the inter-frame interpolation comprises non-linear interpolation.

4. The method of claim 1 , wherein the virtual focus depth of the first key frame provides a tilted virtual focal plane.

5. The method of claim 1 , wherein the virtual focus depth of the first key frame and the virtual focus depth of the second key frame each provide a tilted virtual focal plane.

6. The method of claim 1 , wherein a virtual focus depth of at least one video frame of the first video data provides a tilted virtual focal plane.

7. The method of claim 1 , further comprising inputting the inter-frame interpolation.

8. The method of claim 1 , further comprising generating final output video data by processing the first video data to provide at least one selected from the group consisting of:

compositing;

transitions;

filtering; and

color adjustments

of a plurality of the video frames corresponding to the first video data.

9. A non-transitory system for editing refocusable light field video data, comprising:

first memory, configured to store refocusable light field video data, wherein the refocusable light field video data comprises a plurality of temporally contiguous refocusable light field video frames, wherein each refocusable light field video frame comprises an optical focus;

a user interface, configured to receive input data representative of a first key frame, a second key frame, and inter-frame interpolation; and

processing circuitry, coupled to the first memory and the user interface, configured to:

determine a virtual focus depth for the first key frame, wherein the virtual focus depth for the first key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the first key frame;

determine a virtual focus depth for the second key frame, wherein the virtual focus depth for the second key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the second key frame; and

generate first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus depth for the first key frame, (ii) the virtual focus depth for the second key frame, (iii) an inter-frame interpolation, and (iv) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data comprises a plurality of video frames, each video frame comprising a virtual focus depth which is based on the virtual focus depth for the first key frame, the virtual focus depth for the second key frame, and inter-frame interpolation.

10. The non-transitory system of claim 9 , wherein the user interface further comprises a display configured to visually output a plurality of the refocusable light field video frames.

11. The non-transitory system of claim 10 , wherein the system further comprises:

second memory, coupled to the processing circuitry, configured to store the first video data.

12. The non-transitory system of claim 9 , wherein the processing circuitry is configured to generate final output video by processing the first video data to provide at least one selected from the group consisting of:

compositing;

transitions;

filtering; and

color adjustments

of a plurality of the video frames corresponding to the first video data.

13. The non-transitory system of claim 9 , wherein the inter-frame interpolation comprises linear interpolation.

14. The non-transitory system of claim 9 , wherein the inter-frame interpolation comprises non-linear interpolation.

15. The non-transitory system of claim 9 , wherein the virtual focus depth of the first key frame provides a tilted virtual focal plane.

16. The non-transitory system of claim 9 , wherein a virtual focus depth of at least one video frame of the first video data provides a tilted virtual focal plane.

17. A non-transitory system for editing video refocusable light field video data, comprising:

means for storing refocusable light field video data, wherein the refocusable light field video data comprises a plurality of temporally contiguous refocusable light field video frames, wherein each refocusable light field video frame comprises an optical focus;

means for receiving input data which is representative of a first key frame, a second key frame, and inter-frame interpolation;

means, coupled to the storing and receiving means, for:

determining a virtual focus depth for the first key frame, wherein the virtual focus depth for the first key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the first key frame;

determining a virtual focus depth for the second key frame, wherein the virtual focus depth for the second key frame is different from the optical focus depth of the refocusable light field video frame corresponding to the second key frame; and

generating first video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames using (i) the virtual focus depth for the first key frame, (ii) the virtual focus depth for the second key frame, (iii) an inter-frame interpolation, and (iv) the refocusable light field video data corresponding to the plurality of refocusable light field video frames which are temporally disposed between the first and the second key frames, wherein the first video data comprises a plurality of video frames, each video frame comprising a virtual focus depth which is based on the virtual focus depth for the first key frame, the virtual focus depth for the second key frame, and inter-frame interpolation.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 8, 2019
From: LYTRO, INC.
To: GOOGLE LLC
Reel/Frame 050009/0829 →
SECURITY AGREEMENT Recorded Jun 17, 2013
From: LYTRO, INC.
To: TRIPLEPOINT CAPITAL LLC
Reel/Frame 030623/0434 →
SECURITY AGREEMENT Recorded Jan 31, 2013
From: LYTRO, INC.
To: TRIPLEPOINT CAPITAL LLC
Reel/Frame 029732/0787 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 31, 2012
From: PITTS, COLVIN; NG, YI-REN
To: LYTRO, INC.
Reel/Frame 028691/0482 →
Continuity (5)
Division 12622673 · Nov 20, 2009
Provisional Application 61117621 · Nov 25, 2008
Provisional Application 61120530 · Dec 8, 2008
Provisional Application 61170620 · Apr 18, 2009
Related Publication 20120294590A1 · Nov 22, 2012