IP Library Granted Patent US 11,089,280
Granted Patent B2
US 11,089,280 · App. 15/280,933 · Granted Aug 10, 2021

Apparatus and method for capturing and displaying segmented content

Inventor: Dennis D. Castleman (Fremont, CA)
Assignee: Sony Interactive Entertainment Inc.
H04N13/156G06F3/011G06F3/012G06F3/013G06F3/0482G11B20/10527G11B27/036G11B27/102H04L65/4069H04L65/4076H04L65/4084H04L65/602H04L65/604H04L65/80H04N5/247H04N13/243H04N13/378H04N13/383H04N21/2343H04N21/2387H04N21/4223H04N21/4355H04N21/441H04N21/44218H04N21/4532H04N21/4621H04N21/4728H04N21/482H04N21/6587H04N21/816H04N21/845G11B2020/1062H04N5/23238H04N5/265
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,089,280
App. No.
15/280,933
Granted
Aug 10, 2021
Kind
B2
Abstract

A method for providing content includes determining a viewing direction of a user viewing a content item comprising a plurality of video streams, selecting two or more video streams of the content item based on the viewing direction of the user and directional data associated with the plurality of video streams, decoding the two or more video streams to form two or more decoded video streams, stitching the two or more decoded video streams to form a combined image, and causing the combined image to be displayed to the user. Systems perform similar steps and non-transitory computer readable storage mediums each store one or more computer programs.

Claims (47)

1. A method for providing content comprising:

determining a viewing direction of a user viewing a content item comprising a plurality of video streams;

selecting two or more video streams of the content item based on the viewing direction of the user and directional data associated with the plurality of video streams, wherein the directional data was recorded by a capture system that recorded the content item;

decoding the two or more video streams to form two or more decoded video streams;

stitching the two or more decoded video streams to form a combined image, wherein the stitching comprises selectively stitching overlapping portions of the two or more decoded video streams that fall within a view area of the user when the view area of the user is in a first location and not stitching overlapping portions of the two or more decoded video streams that fall outside of the view area of the user when the view area of the user is in the first location; and

causing the combined image to be displayed to the user.

2. The method of claim 1 , wherein the content item comprises an omnidirectional video.

3. The method of claim 1 , wherein the content item comprises a streaming video and wherein selecting the two or more video streams comprises retrieving only the selected video streams among the plurality of video streams.

4. The method of claim 1 , wherein each of the plurality of video streams is captured by a separate camera in a multi-directional camera assembly.

5. The method of claim 1 , wherein the viewing direction of the user comprises one or more of a head direction and a gaze direction.

6. The method of claim 1 , further comprising: determining the view area based on the viewing direction.

7. The method of claim 6 , wherein decoding the two or more video streams comprises selectively decoding portions of the two or more video streams that fall within the view area.

8. The method of claim 1 , further comprising: adjusting the combined image based on parameters associated with a display device.

9. A system for providing content comprising:

a video decoder;

a graphic processor; and

a central processor configured to:

determine a viewing direction of a user viewing a content item comprising a plurality of video streams;

selectively retrieve two or more video streams of the content item based on the viewing direction of the user and directional data associated with the plurality of video streams, wherein the directional data was recorded by a capture system that recorded the content item;

cause the video decoder to decode the two or more video streams to form two or more decoded video streams;

cause the graphic processor to stitch the two or more decoded video streams to form a combined image, wherein the stitching comprises selectively stitching overlapping portions of the two or more decoded video streams that fall within a view area of the user when the view area of the user is in a first location and not stitching overlapping portions of the two or more decoded video streams that fall outside of the view area of the user when the view area of the user is in the first location; and

cause the combined image to be displayed to the user.

10. The system of claim 9 , wherein the content item comprises an omnidirectional video.

11. The system of claim 9 , wherein the content item comprises a streaming video and wherein selecting the two or more video streams comprises retrieving only the selected video streams among the plurality of video streams.

12. The system of claim 9 , wherein each of the plurality of video streams is captured by a separate camera in a multi-directional camera assembly.

13. The system of claim 9 , wherein the viewing direction of the user comprises one or more of a head direction and a gaze direction.

14. The system of claim 9 , the central processor is further configured to determine the view area based on the viewing direction.

15. The system of claim 14 , wherein the video decoder decodes the two or more video streams by selectively decoding portions of the two or more video streams that fall within the view area.

16. The system of claim 9 , wherein the central processor is further configured to cause the combined image to be adjusted based on parameters associated with a display device.

17. A non-transitory computer readable storage medium storing one or more computer programs configured to cause a processor based system to execute steps comprising:

determining a viewing direction of a user viewing a content item comprising a plurality of video streams;

selecting two or more video streams of the content item based on the viewing direction of the user and directional data associated with the plurality of video streams, wherein the directional data was recorded by a capture system that recorded the content item;

decoding the two or more video streams to form two or more decoded video streams;

stitching the two or more decoded video streams to form a combined image, wherein the stitching comprises selectively stitching overlapping portions of the two or more decoded video streams that fall within a view area of the user when the view area of the user is in a first location and not stitching overlapping portions of the two or more decoded video streams that fall outside of the view area of the user when the view area of the user is in the first location; and

causing the combined image to be displayed to the user.

18. An apparatus for capturing content comprising:

a plurality of cameras forming a multi-directional camera assembly;

at least one inertial measurement unit coupled to the multi-directional camera assembly;

a video encoder;

a memory device; and

a processor configured to:

cause the plurality of cameras to capture a plurality of video streams;

record directional data for the plurality of video streams based on the at least one inertial measurement unit;

cause the video encoder to separately encode each of the plurality of video streams; and

store, in the memory device, a plurality of encoded video streams and the directional data for playback by a playback device;

wherein the playback device is configured to selectively stitch overlapping portions of two or more decoded video streams that fall within a view area of a user when the view area of the user is in a first location and not stitch overlapping portions of the two or more decoded video streams that fall outside of the view area of the user when the view area of the user is in the first location.

19. The apparatus of claim 18 , further comprising: a communication device configured to stream the plurality of encoded video streams and the directional data to a server accessible by the playback device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 7, 2016
From: CASTLEMAN, DENNIS D.
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 039969/0196 →
Continuity (3)
Provisional Application 62357259 · Jun 30, 2016
Provisional Application 62374687 · Aug 12, 2016
Related Publication 20180007339A1 · Jan 4, 2018
Cited By (2)
US 12,363,353 US 12,445,725