Shuffle-based request buffer for managing large request volumes
Approaches are disclosed for managing aspects of content delivery in a multi-tenant environment. A request buffer can be used to remove correlations between requests and randomly shuffle requests without storing all the requests concurrently. A shuffle sharding algorithm can be used to randomly allocate a subset of resources to different users in order to ensure less than a maximum risk of one user impacting the use of all resources allocated to other users. In some embodiments, separate fleets of resources can be maintained for manifests and video segments to allow for more accurate scaling and customization. Multiple manifests can also be associated with a single endpoint to allow multiple media players to obtain similar content segments from the single endpoint.
1. A computer-implemented method, comprising:
storing a plurality of video segments to a plurality of storage devices in a multi-tenant environment;
determining that a rate of requests being received with respect to the plurality of storage requests satisfies at least one buffering criterion;
receiving a request to perform an operation with respect to a specified segment stored to a respective device of the plurality of storage devices;
generating a random number within a range of a number of elements of a request queue;
causing, if it is determined that a previously-received request is stored to the element of the request queue corresponding to the random number, the previously-received request to be transmitted to a corresponding device of the plurality of storage devices storing a segment corresponding to the previously-received request; and
storing the received request to the element of the request queue corresponding to the random number.
2. The computer-implemented method of claim 1 , further comprising:
after the rate of requests being received no longer satisfies at least one buffering criterion, transmitting any remaining requests in the request queue to the plurality of storage devices.
3. The computer-implemented method of claim 1 , wherein the request to perform an operation is a read request, a write request, or a delete request for the specified segment.
4. The computer-implemented method of claim 1 , wherein the at least one buffering criterion corresponds to a request rate threshold, a request volume threshold, or a request traffic pattern.
5. The computer-implemented method of claim 1 , further comprising:
storing a plurality of placeholders to the elements of the request queue, the placeholders being deleted from the request buffer when received requests are to be stored to the elements to which the placeholders are stored.
6. A computer-implemented method, comprising:
receiving a request for access to a data instance stored to one of a plurality of storage locations;
selecting a random element of a request buffer to which to temporarily store the request;
determining that a previously-received request, for access to a second data instance stored to one of the plurality of storage locations, is stored to the random element;
causing the previously-received request to be pulled from the request buffer and processed to determine whether to provide access to the second data instance; and
storing the request to the random element of the request buffer.
7. The computer-implemented method of claim 6 , wherein the random element is determined using a random number, generated using a random number generator, from within a range of elements of the request buffer to which requests are able to be stored.
8. The computer-implemented method of claim 6 , wherein a number of requests received with respect to the plurality of storage locations over a period of time exceeds a range of elements of the request buffer to which requests are able to be stored, and wherein an order of transmission of received requests is able to be randomized without storing all the requests in the request buffer at any time.
9. The computer-implemented method of claim 6 , wherein storing the request and the previously-received request to randomly selected elements of the request buffer removes one or more correlations between the request and the previously-received request.
10. The computer-implemented method of claim 6 , wherein the selecting of the random element is performed in response to at least one buffering criterion being satisfied with respect to the plurality of storage locations.
11. The computer-implemented method of claim 10 , further comprising:
after the at least one buffering criterion is no longer satisfied, transmitting any remaining requests in the request buffer to the plurality of storage locations for processing.
12. The computer-implemented method of claim 6 , wherein the at least one buffering criterion corresponds to a request rate threshold, a request volume threshold, or a request traffic pattern.
13. The computer-implemented method of claim 6 , wherein the request for access is a read request, a write request, or a delete request for the data instance.
14. The computer-implemented method of claim 6 , further comprising:
storing a plurality of placeholders to the elements of the request buffer, the placeholders being deleted from the request buffer when received requests are to be stored to the elements to which the placeholders are stored.
15. The computer-implemented method of claim 6 , wherein a number of the elements in the request queue is selected to improve request throughput and storage location utilization.
16. A system, comprising:
a processor; and
a memory device including instructions that, when executed by the processor, cause the processor to:
receive a request for access to a data instance stored to one of a plurality of storage locations;
select a random element of a request buffer to which to temporarily store the request;
determine that a previously-received request, for access to a second data instance stored to one of the plurality of storage locations, is stored to the random element;
cause the previously-received request to be pulled from the request buffer and processed to determine whether to provide access to the second data instance; and
store the request to the random element of the request buffer.
17. The system of claim 16 , wherein the random element is determined using a random number, generated using a random number generator, from within a range of elements of the request buffer to which requests are able to be stored.
18. The system of claim 16 , wherein a number of requests received with respect to the plurality of storage locations over a period of time exceeds a range of elements of the request buffer to which requests are able to be stored, and wherein an order of transmission of received requests is able to be randomized without storing all the requests in the request buffer at any time.
19. The system of claim 16 , wherein storing the request and the previously-received request to randomly selected elements of the request buffer removes one or more correlations between the request and the previously-received request.
20. The system of claim 16 , wherein the selecting of the random element is performed in response to at least one buffering criterion being satisfied with respect to the plurality of storage locations.