Systems and methods for management of virtualization data
Described in detail herein is a method of copying data of one or more virtual machines being hosted by one or more non-virtual machines. The method includes receiving an indication that specifies how to perform a copy of data of one or more virtual machines hosted by one or more virtual machine hosts. The method may include determining whether the one or more virtual machines are managed by a virtual machine manager that manages or facilitates management of the virtual machines. If so, the virtual machine manager is dynamically queried to automatically determine the virtual machines that it manages or that it facilitates management of. If not, a virtual machine host is dynamically queried to automatically determine the virtual machines that it hosts. The data of each virtual machine is then copied according to the specifications of the received indication.
1. A method of classifying data of virtual machines in a heterogeneous computing environment that includes virtual machines and non-virtual machines, wherein the method is performed by one or more computing systems, each computing system having a processor and memory, the method comprising:
accessing, by the one or more computing systems, at least some data stored by a virtual machine;
classifying the at least some data to create classified data of the virtual machine, wherein classifying the at least some data of the virtual machine includes creating metadata associated with the classified data of the virtual machine;
storing at least the metadata in an index, wherein the index represents, at least in part, the classified data of the virtual machine, and
wherein the index also represents, at least in part, classified data of at least one non-virtual machine;
detecting changes to the at least some data stored by the virtual machine, wherein the detecting changes to the at least some data stored on the virtual machine includes accessing a journal file for tracking operations performed on the at least some data stored on the virtual machine;
updating the index in accordance with the detected changes;
storing the updated index;
determining distinct data objects within the at least some data stored by the virtual machine;
indexing, the distinct data objects; and
storing the indexed distinct data objects in a second index.
2. The method of claim 1 , wherein the virtual machine stores data in a file residing on a filesystem of a virtual machine host hosting the virtual machine, wherein detecting changes to the at least some data stored by the virtual machine includes detecting a modification to the file, and wherein the method further comprises:
accessing the file, wherein the file includes multiple data objects and wherein at least one of the multiple data objects has been modified;
determining the at least one data object that has been modified;
accessing or creating metadata associated with the at least one data object in accordance with the modification;
updating the index with the accessed or created metadata associated with the at least one data object; and
storing the updated index.
3. The method of claim 1 , further comprising:
for at least one of the distinct data objects:
generating a substantially unique identifier for the distinct data object;
determining, based on the substantially unique identifier, if an instance of the distinct data object has already been stored on a single instance storage device; and
if an instance of the distinct data object has not already been stored on the single instance storage device, then storing the data object on the single instance storage device.
4. The method of claim 1 further comprising: automatically determining multiple virtual machines hosted by a virtual machine host;
determining a threshold number of simultaneous storage operations that can be performed upon data of the multiple virtual machines hosted by the virtual machine host; and
performing multiple storage operations upon the data of the multiple virtual machines hosted by the virtual machine host,
wherein the number of simultaneous copy operations performed upon the data of the multiple virtual machines hosted by the virtual machine host does not exceed the threshold number.
5. The method of claim 1 further comprising monitoring for changes to the at least some data stored by the virtual machine.
6. At least one non-transitory computer-readable medium carrying instructions, which when executed by at least one data processor, executes operations to classify data of virtual machines in a heterogeneous computing environment that includes virtual machines and non-virtual machines, the operations comprising:
accessing, by one or more computing systems, at least some data stored by a virtual machine;
classifying the at least some data to create classified data of the virtual machine, wherein classifying the at least some data of the virtual machine includes accessing or creating metadata associated with the classified data of the virtual machine;
storing at least the metadata in an index, wherein the index represents, at least in part, the classified data of the virtual machine, and
wherein the index also represents, at least in part, classified data of at least one non-virtual machine;
detecting changes to the at least some data stored by the virtual machine, wherein detecting changes to the at least some data stored on the virtual machine includes accessing a journal file for tracking operations performed on the at least some data stored on the virtual machine;
accessing or creating metadata in accordance with the detected changes;
updating the index with the accessed or created metadata;
storing the updated index;
determining distinct data objects within the at least some data stored by the virtual machine;
indexing, the distinct data objects; and
storing the indexed distinct data objects in a second index.
7. The non-transitory medium of claim 6 , wherein the virtual machine stores data in a file residing on a filesystem of a virtual machine host hosting the virtual machine, wherein detecting changes to the at least some data stored by the virtual machine includes detecting a modification to the file, and wherein the method further comprises:
accessing the file, wherein the file includes multiple data objects and wherein at least one of the multiple data objects has been modified;
determining the at least one data object that has been modified;
accessing or creating metadata associated with the at least one data object in accordance with the modification;
updating the index with the accessed or created metadata associated with the at least one data object; and
storing the updated index.
8. The non-transitory medium of claim 6 , further comprising:
for at least one of the distinct data objects:
generating a substantially unique identifier for the distinct data object;
determining, based on the substantially unique identifier, if an instance of the distinct data object has already been stored on a single instance storage device; and
if an instance of the distinct data object has not already been stored on the single instance storage device, then storing the data object on the single instance storage device.
9. The non-transitory medium of claim 6 , further comprising automatically determining multiple virtual machines hosted by a virtual machine host;
determining a threshold number of simultaneous storage operations that can be performed upon data of the multiple virtual machines hosted by the virtual machine host; and
performing multiple storage operations upon the data of the multiple virtual machines hosted by the virtual machine host,
wherein the number of simultaneous copy operations performed upon the data of the multiple virtual machines hosted by the virtual machine host does not exceed the threshold number.
10. The non-transitory medium of claim 6 , further comprising monitoring for changes to the at least some data stored by the virtual machine.
11. The non-transitory medium of claim 6 , wherein updating the index is performed by a journaling agent, the journaling agent includes a virtual filter driver module.
12. The non-transitory medium of claim 11 , wherein the journaling agent is deployed on a virtual I/O port or data stack.
13. The non-transitory medium of claim 12 , wherein the journaling agent operates in conjunction with a virtual file management module to record operations executed on the virtual data.
14. A system for classifying data of virtual machines in a heterogeneous computing environment that includes virtual machines and non-virtual machines the system comprising:
one or more computing systems, each computing system having a processor and memory, the one or more computing systems configured to:
access at least some data stored by a virtual machine;
classify the at least some data to create classified data of the virtual machine;
store at least the metadata in an index, wherein the index represents, at least in part, the classified data of the virtual machine, and at least in part, classified data of at least one non-virtual machine;
detect changes to the at least some data stored by the virtual machine, wherein the detecting changes to the at least some data stored on the virtual machine includes accessing a journal file that tracks operations performed on the at least some data stored on the virtual machine;
update the index in accordance with the detected changes;
store the updated index;
determine distinct data objects within the at least some data stored by the virtual machine;
index the distinct data objects; and
store the indexed distinct data objects in a second index.