IP Library › Granted Patent US 11,653,040
Granted Patent B2
US 11,653,040 · App. 16/458,630 · Granted May 16, 2023

Method for audio and video just-in-time transcoding

Inventors: Jon Dahl (San Francisco, CA); Adam Brown (San Francisco, CA); Nick Chadwick (San Francisco, CA); Matt Ward (San Francisco, CA); Scott Kidder (San Francisco, CA)
Assignee: Mux, Inc.
H04N21/234309H04N19/136H04N19/40H04N19/59H04N21/2187H04N21/2393H04N21/23106H04N21/2407H04N21/8456
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,653,040
App. No.
16/458,630
Granted
May 16, 2023
Kind
B2
Abstract

A method for streaming an audio-video file can include: receiving a request for a playback segment of the audio-video file in a rendition from a computational device; in response to identifying absence of the playback segment in the rendition from a rendition cache and identifying absence of an assignment to transcode the playback segment in the rendition: assigning a worker to transcode the playback segment in the rendition. The method can also include, at the worker: identifying a subset of mezzanine segments in the set of mezzanine segments coinciding with a playback interval in the audio-video file; and for each mezzanine segment in the subset of mezzanine segments: concurrently transcoding the mezzanine segment into a rendition segment in the rendition and transmitting the rendition segment coinciding with the playback interval to the computational device via a peer-to-peer stream; and storing the rendition segment in the rendition cache.

Claims (110)

1. A method for streaming an audio-video file comprising:

storing a set of mezzanine segments of the audio-video file, each mezzanine segment in the set of mezzanine segments comprising a portion of the audio-video file;

generating a manifest file for the audio-video file representing:

a first rendition of the audio-video file characterized by a first bitrate and a first resolution; and

a second rendition of the audio-video file characterized by a second bitrate less than the first bitrate and a second resolution less than the first resolution;

prior to transcoding the audio-video file into the first rendition and prior to transcoding the audio-video file into the second rendition, transmitting the manifest file to an audio-video player;

receiving a first request for a playback segment in the first rendition from the audio-video player based on the manifest file;

in response to receiving the first request for the playback segment in the first rendition:

in response to identifying absence of the playback segment in the first rendition from a rendition cache and in response to identifying absence of an assignment to transcode the playback segment in the first rendition:

transcoding the playback segment into the first rendition from the set of mezzanine segments at a first worker;

initiating a first stream between the first worker and the audio-video player; and

storing the playback segment in the first rendition in the rendition cache;

while transcoding the playback segment into the first rendition at the first worker, anticipating a change from the first rendition to the second rendition in a second request from the audio-video player by:

identifying a subsequent segment subsequent to the playback segment in the set of mezzanine segments;

transcoding the subsequent segment into the second rendition from the set of mezzanine segments at a second worker; and

storing the subsequent segment in the second rendition in the rendition cache; and

in response to receiving the second request for the subsequent segment in the second rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the second rendition to the audio-video player from the rendition cache.

2. The method of claim 1 , further comprising:

receiving a third request for the playback segment in the first rendition from a second audio-video player; and

in response to identifying that the playback segment in the first rendition is stored in the rendition cache, transmitting the playback segment in the first rendition to the second audio-video player from the rendition cache.

3. The method of claim 1 , further comprising:

receiving a third request for the playback segment in the first rendition from a second audio-video player; and

in response to identifying that the playback segment is being transcoded into the first rendition:

identifying the first worker transcoding the playback segment;

initiating a second stream between the second audio-video player and the first worker; and

at the first worker, while transcoding the playback segment into the first rendition, transmitting the playback segment in the first rendition to the second audio video player via the second stream.

4. The method of claim 1 , further comprising:

in response to receiving the first request for the playback segment in the first rendition:

transcoding the subsequent segment into the first rendition from the set of mezzanine segments; and

storing the subsequent segment in the first rendition in the rendition cache; and

in response to receiving a third request for the subsequent segment in the first rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the first rendition to the audio-video player from the rendition cache.

5. The method of claim 1 :

wherein generating the manifest file for the audio-video file comprises generating the manifest file for the audio-video file representing:

the first rendition of the audio-video file characterized by the first bitrate and the first resolution;

the second rendition of the audio-video file characterized by the second bitrate less than the first bitrate and the second resolution less than the first resolution; and

a third rendition of the audio-video file characterized by a third bitrate greater than the first bitrate and a third resolution greater than the first resolution;

further comprising, in response to receiving the first request for the playback segment in the first rendition:

transcoding the subsequent segment into the third rendition from the set of mezzanine segments at a third worker; and

storing the subsequent segment in the third rendition in the rendition cache; and

further comprising, in response to receiving a third request for the subsequent segment in the third rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the third rendition to the first audio-video player from the rendition cache.

6. The method of claim 1 , further comprising:

identifying a set of keyframe timestamps in the audio-video file; and

segmenting the audio-video file to define the set of mezzanine segments based on the set of keyframe timestamps.

7. The method of claim 1 , further comprising:

recording, for subsequent requests, viewing data for the playback segment; and

in response to the viewing data of the playback segment failing to satisfy a set of retention criteria, deleting the playback segment in the first rendition from the rendition cache.

8. The method of claim 1 , further comprising:

recording, for subsequent requests, viewing data associated with the first rendition; and

in response to the viewing data associated with the first rendition failing to satisfy a set of retention criteria for the first rendition, deleting the playback segment in the first rendition from the rendition cache.

9. The method of claim 1 , further comprising, at the audio video player, in response to a deterioration in viewing conditions:

adapting a first requested bitrate and a first requested resolution to the second bitrate and the second resolution; and

transmitting the second request for the subsequent segment in the second rendition.

10. The method of claim 1 , wherein, in response to receiving the second request for the subsequent segment in the second rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the second rendition to the audio-video player from the rendition cache comprises:

receiving the second request for the subsequent segment in the second rendition from the audio-video player; and

in response to identifying that the subsequent segment in the second rendition is stored in the rendition cache, transmitting the subsequent segment in the second rendition to the audio-video player from the rendition cache.

11. A method for streaming an audio-video file comprising:

storing a set of mezzanine segments of the audio-video file, each mezzanine segment in the set of mezzanine segments comprising a portion of the audio-video file;

generating a manifest file for the audio-video file representing:

a first rendition of the audio-video file characterized by a first bitrate and a first resolution; and

a second rendition of the audio-video file characterized by a second bitrate greater than the first bitrate and a second resolution greater than the first resolution;

prior to transcoding the audio-video file into the first rendition and prior to transcoding the audio-video file into, the second rendition, transmitting the manifest file to an audio-video player;

receiving a first request for a playback segment in the first rendition from the audio-video player based on the manifest file;

in response to receiving the first request for the playback segment in the first rendition:

in response to identifying absence of the playback segment in the first rendition from a rendition cache and in response to identifying absence of an assignment to transcode the playback segment in the first rendition:

transcoding the playback segment into the first rendition from the set of mezzanine segments at a first worker;

initiating a stream between the first worker and the audio-video player;

storing the playback segment in the first rendition in the rendition cache;

while transcoding the playback segment into the first rendition at the first worker, anticipating a change from the first rendition to the second rendition in a second request from the audio-video player by:

identifying a subsequent segment to the playback segment in the set of mezzanine segments;

transcoding the subsequent segment into the second rendition from the set of mezzanine segments at a second worker; and

storing the subsequent segment in the second rendition in the rendition cache; and

in response to receiving the second request for the subsequent segment in the second rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the second rendition to the audio-video player from the rendition cache.

12. The method of claim 11 :

wherein generating the manifest file for the audio-video file comprises generating the manifest file for the audio-video file representing:

the first rendition of the audio-video file characterized by the first bitrate and the first resolution;

the second rendition of the audio-video file characterized by the second bitrate greater than the first bitrate and the second resolution greater than the first resolution; and

a third rendition of the audio-video file characterized by a third bitrate less than the first bitrate and a third resolution less than the first resolution;

further comprising, in response to receiving the first request for the playback segment in the first rendition:

transcoding the subsequent segment into the third rendition from the set of mezzanine segments at a third worker; and

storing the subsequent segment in the third rendition in the rendition cache; and

further comprising, in response to receiving a third request for the subsequent segment in the third rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the third rendition to the audio-video player from the rendition cache.

13. The method of claim 11 , further comprising, at the audio video player, in response to a deterioration in viewing conditions:

adapting the first bitrate and the first resolution to the second bitrate and the second resolution; and

transmitting the second request for the subsequent segment in the second rendition.

14. The method of claim 11 , wherein, in response to receiving the second request for the subsequent segment in the second rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the second rendition to the audio-video player from the rendition cache comprises:

receiving the second request for the subsequent segment in the second rendition from the audio-video player; and

in response to identifying that the subsequent segment in the second rendition is stored in the rendition cache, transmitting the subsequent segment in the second rendition to the audio-video player from the rendition cache.

15. The method of claim 11 , further comprising:

receiving a second request for the playback segment in the first rendition from a second audio-video player; and

in response to identifying that the playback segment in the first rendition is stored in the rendition cache, transmitting the playback segment in the first rendition to the second audio-video player from the rendition cache.

16. The method of claim 11 , further comprising:

receiving a second request for the playback segment in the first rendition from a second audio-video player; and

in response to identifying that the playback segment is being transcoded into the first rendition:

identifying the first worker transcoding the playback segment;

initiating a second stream between the second audio-video player and the first worker; and

at the first worker, while transcoding the playback segment, transmitting the playback segment in the first rendition to the second audio video player via the second stream.

17. The method of claim 11 , further comprising:

in response to receiving the first request for the playback segment in the first rendition:

transcoding the subsequent segment into the first rendition from the set of mezzanine segments at a third worker; and

storing the subsequent segment in the first rendition in the rendition cache; and

in response to receiving a third request for the subsequent segment in the first rendition from the audio-video player based on the manifest file, transmitting the subsequent segment in the first rendition to the audio-video player from the rendition cache.

18. The method of claim 11 , further comprising:

identifying a set of keyframe timestamps in the audio-video file; and

segmenting the audio-video file to define the set of mezzanine segments based on the set of keyframe timestamps.

19. The method of claim 11 , further comprising:

recording, for subsequent requests, viewing data for the playback segment; and

in response to the viewing data of the playback segment failing to satisfy a set of retention criteria, deleting the playback segment in the first rendition from the rendition cache.

20. The method of claim 11 , further comprising:

recording, for subsequent requests, viewing data associated with the first rendition; and

in response to the viewing data associated with the first rendition failing to satisfy a set of retention criteria for the first rendition, deleting the playback segment in the first rendition from the rendition cache.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 24, 2020
From: DAHL, JON; BROWN, ADAM; CHADWICK, NICK; WARD, MATT; KIDDER, SCOTT
To: MUX, INC.
Reel/Frame 053309/0439 →
Continuity (2)
Provisional Application 62694408 · Jul 5, 2018
Related Publication 20200037014A1 · Jan 30, 2020