Scalable image distribution in virtualized server environments
A method and system include replicating an image representing a sequence of bytes on a local storage medium on a target device by determining a similarity between images and reconstructing a new image using equivalent blocks from one or more similar images locally available on the target device or available on donor devices to reduce network link usage and transfer time in replicating the image.
1. A method, comprising:
replicating an image representing a sequence of bytes on a local storage medium on a target device by:
determining a similarity between images; and
reconstructing a new image by copying data from equivalent blocks from one or more similar images locally available on the target device or available on a best fit donor devices if no similar images are locally available, to reduce network link usage and transfer time in replicating the image, wherein the new image represents a virtual machine (VM).
2. The method as recited in claim 1 , wherein determining a similarity includes performing a checksum on a byte-by-byte comparison between two blocks of images with a same hashcode.
3. The method as recited in claim 1 , wherein the similarity is stored in an image dependency matrix denoting a fraction of blocks that are identical between two images.
4. The method as recited in claim 3 , wherein the image dependency matrix includes an index for identical blocks between the two images.
5. The method as recited in claim 1 , wherein reconstructing includes querying donor devices for the equivalent blocks which are not available in local storage but are needed to reconstruct the image.
6. The method as recited in claim 1 , wherein reconstructing includes accessing a storage server to find remaining blocks needed to reconstruct the new image.
7. The method as recited in claim 1 , wherein reconstructing the new image includes creating a copy-on-write copy of the one or more similar images.
8. The method as recited in claim 1 , wherein the best fit donor device for reconstructing the new image is determined by evaluating at least one criteria for a plurality of donors from a set of qualified donors.
9. The method as recited in claim 8 , wherein the criteria includes at least one of bandwidth, utilization, security, latency, or cost.
10. The method as recited in claim 8 , wherein querying includes providing an overlay network to search for donors.
11. The method as recited in claim 1 , wherein the reconstructing includes creating a new VM instance or migrating an existing VM instance.
12. The method as recited in claim 11 , further comprising starting the new VM without all of the blocks of the new VM being locally available.
13. A method, comprising:
determining a similarity between images in a distributed network;
determining an image dependency matrix denoting a fraction of blocks that are identical between pairs of images;
searching local storage for blocks to reconstruct a new image using the image dependency matrix;
if all blocks needed for the new image are not in local storage, querying donor hosts for the blocks to determine a best fit donor host for reconstructing the new image using the image dependency matrix and copying data from the fraction of blocks that are identical between pairs of images; and
if all blocks needed for the new VM are not in available from the donor hosts, accessing a storage server to find remaining blocks needed to reconstruct the new image, wherein the new image represents a virtual machine (VM).
14. The method as recited in claim 13 , wherein determining a similarity includes performing a checksum on a byte-by-byte comparison between two images with a same hashcode.
15. The method as recited in claim 13 , wherein determining an image dependency matrix includes storing an index for identical blocks between the pairs of images.
16. The method as recited in claim 13 , wherein searching local storage includes creating a copy-on-write copy of a locally available image to construct the new image.
17. The method as recited in claim 13 , wherein the best fit donor device for reconstructing the new image is determined by evaluating at least one criteria for a plurality of host donors from a set of qualified host donors.
18. The method as recited in claim 17 , wherein the criteria includes at least one of bandwidth, utilization, latency or cost.
19. The method as recited in claim 17 , wherein querying includes providing an overlay network to search for donor hosts.
20. The method as recited in claim 13 , wherein the method includes creating a new VM instance or migrating an existing VM instance.
21. The method as recited in claim 20 , further comprising starting the new VM without all of the blocks of the new VM being locally available.