Online data format conversion
In some examples, a data management system generates snapshots in a distributed file system based on a protocol or a user triggered event, The data management system identifies a snappable file in a distributed file system and a first data block in the snappable file, the first data block including data and attribute data. The system scans an index file to access the attribute data of the first data block and initiates construction of a patch file based on the accessed attribute data. The system repeats the scanning of the index file to access attribute data of at least a further second data block, the second data block including data and attribute data, and completes construction of the patch file based on the accessed attribute data of the first and second data blocks. The system generates conversion simulation information by collecting attribute data for all the data blocks of the constructed patch file, and writes the simulation information to a patch file image.
1. A method of online data conversion, the method including:
identifying a snappable file in a distributed file system;
identifying a first data block in the snappable file, the first data block including first data, wherein first attribute data associated with the first data block is stored in an index file associated with the snappable file;
scanning the index file to access the first attribute data of the first data block;
constructing a virtual patch file based on the accessed first attribute data, wherein constructing the virtual patch file comprises writing the first attribute data to the virtual patch file, refraining from reading data from the first data block, and refraining from writing the first data block to the virtual patch file;
repeating the scanning of the index file to access second attribute data of at least a further second data block in the snappable file, the second data block including second data;
completing construction of the virtual patch file based on the accessed second attribute data, wherein completing the construction of the virtual patch file comprises writing the second attribute data to the virtual patch file, refraining from reading data from the second data block, and refraining from writing the second data block to the virtual patch file;
generating conversion simulation information by collecting third attribute data associated with the constructed virtual patch file, the first attribute data for the first data block, and the second attribute data for the second data block used to construct the virtual patch file; and
writing the generated conversion simulation information to a patch file image.
2. The method of claim 1 , wherein the first and second attribute data of the first and second data blocks includes at least logical space offset and data size information.
3. The method of claim 1 , further comprising:
receiving a request to transfer data of the snappable file to a remote location, the transfer involving or necessitating a conversion of data from a first data format to a second data format; and
effecting a data format conversion for the transfer using only the conversion simulation information.
4. The method of claim 1 , further comprising:
receiving a read request for the first data in the first data block or the second data in the second data block; and
re-routing the read request to corresponding data in a journaled patch file using information contained in the patch file image.
5. A data management system to generate snapshots in a distributed file system based on a protocol or a user triggered event, the data management system comprising at least one processor configured to perform operations including:
identifying a snappable file in a distributed file system;
identifying a first data block in the snappable file, the first data block including first data, wherein first attribute data associated with the first data block is stored in an index file associated with the snappable file;
scanning the index file to access the first attribute data of the first data block;
constructing a virtual patch file based on the accessed first attribute data, wherein constructing the virtual patch file comprises writing the first attribute data to the virtual patch file, refraining from reading data from the first data block, and refraining from writing the first data block to the virtual patch file;
repeating the scanning of the index file to access second attribute data of at least a further second data block in the snappable file, the second data block including second data;
completing construction of the virtual patch file based on the accessed second attribute data, wherein completing the construction of the virtual patch file comprises writing the second attribute data to the virtual patch file, refraining from reading data from the second data block, and refraining from writing the second data to the virtual patch file;
generating conversion simulation information by collecting third attribute data associated with the constructed virtual patch file, the first attribute data for the first data block, and the second attribute data for the second data block used to construct the virtual patch file; and
writing the generated conversion simulation information to a patch file image.
6. The system of claim 5 , wherein the first and second attribute data of the first and second data blocks includes at least logical space offset and data size information.
7. The system of claim 5 , wherein the operations further comprise:
receiving a request to transfer data of the snappable file to a remote location, the transfer involving or necessitating a conversion of data from a first data format to a second data format; and
effecting a data format conversion for the transfer using only the conversion simulation information.
8. The system of claim 5 , wherein the operations further comprise:
receiving a read request for the first data in the first data block or the second data in the second data block; and
re-routing the read request to corresponding data in a journaled patch file using information contained in the patch file image.
9. A non-transitory machine-readable medium including instructions which, when read by a machine, cause a machine to perform operations in a method of generating snapshots of a distributed file system, the operations including:
identifying a snappable file in a distributed file system;
identifying a first data block in the snappable file, the first data block including first data, wherein first attribute data associated with the first data block is stored in an index file associated with the snappable file;
scanning the index file to access the first attribute data of the first data block;
constructing a virtual patch file based on the accessed first attribute data, wherein constructing the virtual patch file comprises writing the first attribute data to the virtual patch file, refraining from reading data from the first data block, and refraining from writing the first data block to the virtual patch file;
repeating the scanning of the index file to access second attribute data of at least a further second data block in the snappable file, the second data block including second data;
completing construction of the virtual patch file based on the accessed second attribute data, wherein completing the construction of the virtual patch file comprises writing the second attribute data to the virtual patch file, refraining from reading data from the second data block, and refraining from writing the second data block to the virtual patch file;
generating conversion simulation information by collecting third attribute data associated with the constructed virtual patch file, the first attribute data for the first data block, and the second attribute data for the second data block used to construct the virtual patch file; and
writing the generated conversion simulation information to a patch file image.
10. The medium of claim 9 , wherein the first and second attribute data of the first and second data blocks includes at least logical space offset and data size information.
11. The medium of claim 9 , wherein the operations further comprise:
receiving a request to transfer data of the snappable file to a remote location, the transfer involving or necessitating a conversion of data from a first data format to a second data format; and
effecting a data format conversion for the transfer using only the conversion simulation information.
12. The medium of claim 9 , wherein the operations further comprise:
receiving a read request for the first data in the first data block or the second data in the second data block; and
re-routing the read request to corresponding data in a journaled patch file using information contained in the patch file image.