Dedupe file system for bulk data migration to cloud platform
In one aspect, a computer-implemented method useful for bulk data migration with a dedupe file system includes the step of providing a data-transfer appliance, wherein the data-transfer appliance comprises a specified number of hard drive data disks configured in a specified number of RAID (redundant array of independent disks) volumes in each RAID group. The computer-implemented method includes the step of creating a single dedupe mountpoint on the data-transfer appliance for a dedupe process, wherein the dedupe process communicates only with the single dedupe mountpoint. The computer-implemented method includes the step of creating a single namespace of all RAID volumes under the single dedupe mountpoint. The computer-implemented method includes the step of creating a scalable namespace for deduplication without a volume manager. The computer-implemented method includes the step of providing n-number of writer threads, wherein each writer-thread works with one RAID group assignment and the writer-threads are evenly distributed.
1. A computer-implemented method useful for bulk data migration with a dedupe file system comprising:
providing a data-transfer appliance, wherein the data-transfer appliance comprises a specified number of hard drive data disks configured in a specified number of RAID (redundant array of independent disks) volumes in each individual RAID group, wherein the individual RAID group comprises at least one directory as a mountpoint of the individual RAID group, and
wherein the directory has a flat hierarchy;
creating a single dedupe mountpoint on the data-transfer appliance for a dedupe process,
wherein the dedupe process communicates only with the single dedupe mountpoint;
creating a single scalable namespace for all RAID volumes under the single dedupe mountpoint;
wherein the single scalable namespace for deduplication is without a logical volume manager; and
providing a number of writer threads, wherein each writer thread works with one RAID group assignment and the writer threads are evenly distributed across all RAID groups,
wherein the writer thread comprises a thread in the dedupe process that listens on a socket for a plain text data and dedupes the plain text data, and
wherein the writer thread encrypts and compresses the deduped plain text data before writing the deduped plain text data to a dedupe storage in the data-transfer appliance.
2. The computer-implemented method of claim 1 , wherein the specified number of hard drive data disks comprises sixty (60) of the hard drive data disks.
3. The computer-implemented method of claim 2 , wherein the RAID configurations comprises a number of RAID6 groups, each of fifteen (15) disks.
4. The computer-implemented method of claim 3 , wherein each of the RAID6 groups comprises an ext4 file system.
5. The computer-implemented method of claim 3 , wherein the RAID configurations comprises four (4) of the RAID6 groups, each of fifteen disks.
6. The computer-implemented method of claim 1 , wherein the even distribution of the writer threads across the RAID6 groups provide storage usage across all the RAID groups.
7. A computer system comprising:
a processor configured to execute the instructions;
a memory containing the instructions when executed on the processor, causes the processor to perform operations that:
provide a data-transfer appliance, wherein the data-transfer appliance comprises a specified number of hard drive data disks configured in a specified number of RAID (redundant array of independent disks) volumes in each RAID group, wherein each individual RAID group comprises at least one directory as a mountpoint of the individual RAID group, and wherein the directory has a flat hierarchy;
create a single dedupe mountpoint on the data-transfer appliance for a dedupe process, wherein the dedupe process communicates only with the single dedupe mountpoint;
create a single scalable namespace for all of the RAID volumes under the single dedupe mountpoint, wherein the single scalable namespace for deduplication is without a logical volume manager; and
provide a number of writer threads, wherein each writer thread works with one RAID group assignment and the writer threads are evenly distributed across all RAID groups,
wherein the writer thread comprises a thread in the dedupe process that listens on a socket for a plain text data and dedupes the plain text data, and
wherein the writer thread encrypts and compresses the deduped plain text data before writing the deduped plain text data to a dedupe storage in the data-transfer appliance.
8. The computer system of claim 7 , wherein the specified number of hard drive data disks comprises sixty (60) of the hard drive data disks.
9. The computer system of claim 8 , wherein the RAID configurations comprises a number of RAID6 groups, each of fifteen (15) disks.
10. The computer system of claim 9 , wherein each of the RAID6 groups comprises an ext4 file system.
11. The computer system of claim 10 , wherein the RAID configurations comprises four (4) RAID6 groups, each of fifteen disks.
12. The computer system of claim 11 , wherein the data-transfer appliance comprises an offline data-transfer appliance with one petabyte of storage.
13. The computer system of claim 12 , wherein the offline data-transfer appliance comprises an operating system and one or more central processing units (CPU).
14. The computer system of claim 13 ,
wherein the offline data-transfer appliance performs link aggregation and assigns itself an internet-protocol (IP) address, and
wherein the even distribution of the writer threads across the RAID6 groups provide storage usage across all the RAID6 groups.
15. The computer system of claim 7 , further comprising:
with the writer thread, appends a transformed data (TD) until a size of a dedupe segment file is greater than a specified size.
16. The computer system of claim 15 , wherein the specified size of the dedupe segment file is greater than eight gigabytes (8 GB).