Asynchronous input and output for snapshots of virtual machines
A data management system having a storage appliance configured to store a snapshot of a virtual machine; and one or more processors in communication with the storage appliance. The one or more processors are configured to perform operations including: identifying a plurality of shards of the virtual machine; requesting a shard snapshot of each of the plurality of shards; receiving the shard snapshots asynchronously; ordering the received shard snapshots sequentially into a results queue; and storing a single snapshot of the virtual machine based on the ordered shard snapshots. The operations may further include maintaining a flow control queue that limits a number of the requested shard snapshots.
1. A computer-implemented method at a data management system, the method comprising:
sharding an entire virtual machine into a plurality of shards, wherein each of the shards corresponds to a different portion of the entire virtual machine;
transmitting separate requests for shard snapshots corresponding to at least a first subset of shards of the plurality of shards;
receiving, in response to the separate requests for the shard snapshots, the requested shard snapshots out of order;
maintaining an offset-slot mapping indicating ordering of the requested shard snapshots, wherein the offset-slot mapping uses disk addresses of the entire virtual machine to indicate the ordering of the requested shard snapshots;
storing a shard snapshot that is not maintained in the offset-slot mapping in an early received map, wherein the early received map is configured to hold shard snapshot data until the ordering indicated by the offset-slot mapping is updated based at least in part on a receive queue;
ordering the received out of order shard snapshots and the shard snapshot that is stored in the early received map in order into a results queue using the offset-slot mapping that is maintained for the requested shard snapshots and updated to include the shard snapshot based at least in part on the receive queue; and
storing a single snapshot of the entire virtual machine in a computer memory based at least in part on the ordered shard snapshots.
2. The method of claim 1 , further comprising:
maintaining a flow control queue that limits a number of the requested shard snapshots, wherein the flow control queue stores a set of tokens that limit the number of the requested shard snapshots and receives a token from the receive queue upon storage of a shard snapshot in the results queue.
3. The method of claim 2 , further comprising:
maintaining the receive queue and transferring a token from the flow control queue to the receive queue upon receiving one of the requested shard snapshots of the plurality of shards.
4. The method of claim 3 , further comprising:
updating the maintained offset-slot mapping with ordering of the shard snapshot not in the offset-slot mapping; and
moving the shard snapshot not in the offset-slot mapping into the results queue after the updating.
5. The method of claim 1 , further comprising presenting the ordered shard snapshots in order to a read Application Programming Interface for storage in the computer memory.
6. The method of claim 1 , further comprising enforcing a Secure Sockets Layer in the receiving the requested out of order shard snapshots.
7. A non-transitory, machine-readable medium storing instructions which, when read by a machine, cause the machine to perform operations comprising, at least:
sharding an entire virtual machine into a plurality of shards, wherein each of the shards corresponds to a different portion of the entire virtual machine;
transmitting separate requests for shard snapshots corresponding to at least a first subset of shards of the plurality of shards;
receiving, in response to the separate requests for the shard snapshots, the requested shard snapshots out of order;
maintaining an offset-slot mapping indicating ordering of the requested shard snapshots, wherein the offset-slot mapping uses disk addresses of the entire virtual machine to indicate the ordering of the requested shard snapshots;
storing a shard snapshot that is not maintained in the offset-slot mapping in an early received map, wherein the early received map is configured to hold shard snapshot data until the ordering indicated by the offset-slot mapping is updated based at least in part on a receive queue;
ordering the received out of order shard snapshots and the shard snapshot that is stored in the early received map in order into a results queue using the offset-slot mapping that is maintained for the requested shard snapshots and updated to include the shard snapshot based at least in part on the receive queue; and
storing a single snapshot of the entire virtual machine in a computer memory based at least in part on the ordered shard snapshots.
8. The medium of claim 7 , wherein the operations further include:
maintaining a flow control queue that limits a number of the requested shard snapshots, wherein the flow control queue stores a set of tokens that limit the number of the requested shard snapshots and receives a token from the receive queue upon storage of a shard snapshot in the results queue.
9. The medium of claim 8 , wherein the operations further include:
maintaining the receive queue and transferring a token from the flow control queue to the receive queue upon receiving one of the requested shard snapshots of the plurality of shards.
10. The medium of claim 9 , wherein the operations further include:
updating the maintained offset-slot mapping with ordering of the shard snapshot not in the offset-slot mapping; and
moving the shard snapshot not in the offset-slot mapping into the results queue after the updating.
11. The medium of claim 7 , wherein the operations further include presenting the ordered shard snapshots in order to a read Application Programming Interface for storage in the computer memory.
12. The medium of claim 7 , wherein the operations further include enforcing a Secure Sockets Layer in the receiving the requested out of order shard snapshots.
13. A data management system, comprising:
a computer memory configured to store a snapshot of an entire virtual machine;
one or more processors in communication with the computer memory, the one or more processors configured to perform operations including:
sharding the entire virtual machine into a plurality of shards, wherein each of the shards corresponds to a different portion of the entire virtual machine;
transmitting separate requests for shard snapshots corresponding to at least a first subset of shards of the plurality of shards;
receiving, in response to the separate requests for the shard snapshots, the requested shard snapshots out of order;
maintaining an offset-slot mapping indicating ordering of the requested shard snapshots, wherein the offset-slot mapping uses disk addresses of the entire virtual machine to indicate the ordering of the requested shard snapshots;
storing a shard snapshot that is not maintained in the offset-slot mapping in an early received map, wherein the early received map is configured to hold shard snapshot data until the ordering indicated by the offset-slot mapping is updated based at least in part on a receive queue;
ordering the received out of order shard snapshots and the shard snapshot that is stored in the early received map in order into a results queue using the offset-slot mapping that is maintained for the requested shard snapshots and updated to include the shard snapshot based at least in part on the receive queue; and
storing a single snapshot of the entire virtual machine in the computer memory based at least in part on the ordered shard snapshots.
14. The system of claim 13 , wherein the operations further include:
maintaining a flow control queue that limits a number of the requested shard snapshots, wherein the flow control queue stores a set of tokens that limit the number of the requested shard snapshots and receives a token from the receive queue upon storage of a shard snapshot in the results queue.
15. The system of claim 14 , wherein the operations further include:
maintaining the receive queue and transferring a token from the flow control queue to the receive queue upon receiving one of the requested shard snapshots of the plurality of shards.
16. The system of claim 15 , wherein the operations further include:
updating the maintained offset-slot mapping with ordering of the shard snapshot not in the offset-slot mapping; and
moving the shard snapshot not in the offset-slot mapping into the results queue after the updating.
17. The system of claim 13 , wherein the operations further include presenting the ordered shard snapshots in order to a read Application Programming Interface for storage in the computer memory.
18. The system of claim 13 , wherein the operations further include enforcing a Secure Sockets Layer in the receiving the requested out of order shard snapshots.