System and method for large number of simultaneous read streams for DDVE data stored into object store
One example method includes receiving read requests from a backup application, determining whether or not adequate compute instances are available to service the read requests, when adequate compute instances are not available to service the read requests, spawning additional compute instances so that the read requests can be serviced, servicing the read requests using the additional compute instances, and after the read requests have been serviced, spinning down any compute instances that are no longer needed.
1 . A method, comprising:
receiving read requests, each read request issued by a backup application to retrieve previously stored backup data;
determining, for each read request, whether or not a compute instance dedicated to servicing that read request is available;
when no such compute instance is available for the read request, spawning a lightweight data mover compute instance specifically for that read request;
servicing the read request using only the lightweight data mover compute instance spawned for that read request; and
after completion of the read request, automatically terminating the lightweight data mover compute instance that serviced the read request,
wherein the lightweight data mover compute instance is not reused to service a different read request after termination.
2 . The method as recited in claim 1 , wherein the spawning of the lightweight data mover compute instance is performed automatically based on the read request that has been received.
3 . The method as recited in claim 1 , wherein termination of the lightweight data mover compute instance is performed automatically immediately after the corresponding read request is fully serviced.
4 . The method as recited in claim 1 , wherein the lightweight data mover compute instance is configurable at a time when it is being spawned for the read request.
5 . The method as recited in claim 1 , wherein no lightweight data mover compute instance services more than one read request.
6 . The method as recited in claim 5 , wherein the lightweight data mover compute instance obtains metadata details for a file that has been identified in the read request.
7 . The method as recited in claim 6 , wherein the lightweight data mover compute instance receives data access credentials from the backup application, and the lightweight data mover compute instance uses the data access credentials to obtain the metadata details.
8 . The method as recited in claim 7 , wherein the lightweight data mover compute instance uses the metadata details to read segment data, for a file, from an object store, and the lightweight data mover compute instance decrypts the segment data, decompresses the decrypted segment data, and returns the decrypted and decompressed segment data to the backup application.
9 . The method as recited in claim 1 , wherein multiple lightweight data mover compute instances service different read requests in parallel, each lightweight data mover compute instance servicing only a single read request.
10 . The method as recited in claim 1 , wherein the lightweight data mover compute instance comprises a virtual machine, or a container.
11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving read requests, each read request issued by a backup application to retrieve previously stored backup data;
determining, for each read request, whether or not a compute instance dedicated to servicing that read request is available;
when no such compute instance is available for the read request, spawning a lightweight data mover compute instance specifically for that read request;
servicing the read request using only the lightweight data mover compute instance spawned for that read request; and
after completion of the read request, automatically terminating the lightweight data mover compute instance that serviced the read request,
wherein the lightweight data mover compute instance is not reused to service a different read request after termination.
12 . The non-transitory storage medium as recited in claim 11 , wherein the spawning of the lightweight data mover compute instance is performed automatically based on the read request that has been received.
13 . The non-transitory storage medium as recited in claim 11 , wherein termination of the lightweight data mover compute instance is performed automatically immediately after the read request is fully serviced.
14 . The non-transitory storage medium as recited in claim 11 , wherein the lightweight data mover compute instance is configurable at a time when it is being spawned for the read request.
15 . The non-transitory storage medium as recited in claim 11 , wherein no lightweight data mover compute instance services more than one read request.
16 . The non-transitory storage medium as recited in claim 15 , wherein the lightweight data mover compute instance obtains metadata details for a file that has been identified in the read request.
17 . The non-transitory storage medium as recited in claim 16 , wherein the lightweight data mover compute instance receives data access credentials from the backup application, and the lightweight data mover compute instance uses the data access credentials to obtain the metadata details.
18 . The non-transitory storage medium as recited in claim 17 , wherein the lightweight data mover compute instance uses the metadata details to read segment data, for a file, from an object store, and the lightweight data mover compute instance decrypts the segment data, decompresses the decrypted segment data, and returns the decrypted and decompressed segment data to the backup application.
19 . The non-transitory storage medium as recited in claim 11 , wherein multiple lightweight data mover compute instances service different read requests in parallel, each lightweight data mover compute instance servicing only a single read request.
20 . The non-transitory storage medium as recited in claim 11 , wherein the lightweight data mover compute instance comprises a virtual machine, or a container.