Encoding scalability with brokers
A scalable architecture is provided for decentralized scaling of resources in a media content encoding platform. The scalable architecture is comprised of a first slicing tier, a second broker tier, and a third encoding tier. Each tier can be horizontally and vertically scaled independent of one another. The second broker tier receives media content slices from the first slicing tier. The second broker tier retains the slices directly in main memory of different brokers without writing the slices to a database or disk. The brokers distribute the slices from main memory across the third encoding tier for encoding based on availability of different encoders in the third tier. This architecture improves overall encoding performance as some of the delays associated with managing and distributing the slices at the second tier are eliminated by operation of the brokers.
1. A method comprising:
receiving a plurality of slices at a first set of brokers, wherein each slice of the plurality of slices comprises a different segment of at least one media content;
queuing the plurality of slices in memory of the first set of brokers without writing the plurality of slices to disk;
encoding a first set of the plurality of slices with a first set of encoders;
determining a first rate corresponding to a rate of receiving the plurality of slices at the first set of brokers, and a second rate corresponding to a rate of encoding the plurality of slices with the first set of encoders based on said encoding;
scaling from the first set of encoders to a larger second set of encoders in response to the first rate of receiving the plurality of slices being greater than the second rate of encoding the plurality of slices;
encoding a second set of the plurality of slices with the second set of encoders; and
determining a third rate that corresponds to a rate of encoding the plurality of slices with the second set of encoders based on said encoding the second set of slices, and wherein the third rate is greater than the first rate or the second rate.
2. The method of claim 1 , wherein said scaling from the first set of brokers comprises adding at least one broker to the first set of brokers, the at least one broker comprising additional memory that is used in retaining a subset of the plurality of slices.
3. The method of claim 1 further comprising removing the first set of slices from memory of the first set of brokers in response to encoding the first set of slices.
4. The method of claim 1 further comprising assigning a first subset of the first set of encoders to a first broker of the first set of brokers and a different second subset of the second set of encoders to a second broker of the first set of brokers.
5. The method of claim 4 , wherein encoding the first set of slices comprises each encoder of the first subset of encoders encoding slices retained in memory of the first broker and each encoder of the second subset of encoders encoding slices retained in memory of the second broker.
6. A method comprising:
configuring a first set of a plurality of machines as a first set of brokers;
receiving a plurality of slices directly into memory of the first set of brokers without writing the plurality of slices to storage or a database, wherein each slice of the plurality of slices comprises a different segment of at least one media content;
configuring a second set of the plurality of machines as a first set of encoders;
passing a first set of the plurality of slices from the first set of brokers to the first set of encoders based on messaging exchanged between the first set of brokers and the first set of encoders;
encoding the first set of slices with the first set of encoders while retaining a second set of the plurality of slices in memory of the first set of brokers; and
scaling from the first set of brokers and the first set of encoders to a second set of brokers and a second set of encoders by reconfiguring at least one machine from operating as a broker in the first set of brokers to operating as an encoder in the second set of encoders, or by reconfiguring at least one machine from operating as an encoder in the first set of encoders to operating as a broker in the second set of brokers.
7. The method of claim 6 , further comprising passing a first slice from the second set of slices to a first encoder of the first set of encoders after the first encoder completes encoding of a slice from the first set of slices.
8. The method of claim 6 further comprising removing the first set of slices from said memory of the first set of brokers in response to said encoding.
9. A system comprising:
a plurality of machines comprising:
a first set of the plurality of machines configured as a first set of brokers, the first set of brokers receiving a plurality of slices directly into memory of the first set of brokers without writing the plurality of slices to storage or a database, wherein each slice of the plurality of slices comprises a different segment of at least one media content;
a second set of the plurality of machines configured as a first set of encoders, the first set of encoders (i) receiving a first set of the plurality of slices from the first set of brokers based on messaging exchanged between the first set of brokers and the first set of encoders, and (ii) encoding the first set of slices with the first set of encoders while retaining a second set of the plurality of slices in memory of the first set of brokers; and
a device comprising:
a non-transitory computer-readable medium storing a set of processor-executable instructions; and
one or more processors configured to execute the set of processor-executable instructions, wherein executing the set of processor-executable instructions causes the one or more processors to:
scale from the first set of brokers and the first set of encoders to a second set of brokers and a second set of encoders by reconfiguring at least one machine from operating as a broker in the first set of brokers to operating as an encoder in the second set of encoders, or by reconfiguring at least one machine from operating as an encoder in the first set of encoders to operating as a broker in the second set of brokers.