Optimizing Transfer Of Small Files Of A Storage System

1 . A method comprising:

determining to transfer a dataset stored at a first storage system to a second storage system;

invoking an operation to retrieve, for a plurality of portions of the dataset having a size less than a threshold size, an identifier, metadata, and data stored in the plurality of portions of the dataset; and

transmitting the identifiers, the metadata, and the data for the plurality of portions of the dataset having a size less than the threshold size to the second storage system.

2 . The method of claim 1 , wherein the operation comprises a single remote procedure call to the first storage system.

3 . The method of claim 1 , further comprising:

iteratively invoking the operation for one or more directories in the dataset at the first storage system.

4 . The method of claim 1 , further comprising:

concurrently invoking the operation on directories of the dataset to transfer the dataset over a plurality of communication channels in parallel.

5 . The method of claim 1 , wherein the plurality of portions of the dataset comprises at least one of a plurality of files, a plurality of data objects, or a plurality of blocks of data.

6 . The method of claim 1 , further comprising:

determining, for a plurality of directories in the dataset, a number of portions of the dataset having a size less than the threshold size; and

generating a plurality of data streams to transfer the dataset, wherein the portions of the dataset identified by the operation are distributed among the plurality of data streams based on the number of portions of the dataset in each directory having a size less than the threshold size.

7 . The method of claim 6 , wherein a number of portions of the dataset transmitted via each of the plurality of data streams comprises a substantially equal number of files.

8 . A system comprising:

a memory; and

a processing device operatively coupled with the memory, the processing device configured to:

determine to transfer a dataset stored at a first storage system to a second storage system;

invoke an operation to retrieve, for a plurality of portions of the dataset having a size less than a threshold size, an identifier, metadata, and data stored in the plurality of portions of the dataset; and

transmit the identifiers, the metadata, and the data for the plurality of portions of the dataset having a size less than the threshold size to the second storage system.

9 . The system of claim 8 , wherein the operation comprises a single remote procedure call to the first storage system.

10 . The system of claim 8 , wherein the processing device is further configured to:

iteratively invoke the operation for one or more directories in the dataset at the first storage system.

11 . The system of claim 8 , wherein the processing device is further configured to:

concurrently invoke the operation on directories of the dataset to transfer the dataset over a plurality of communication channels in parallel.

12 . The system of claim 8 , wherein the plurality of portions of the dataset comprises at least one of a plurality of files, a plurality of data objects, or a plurality of blocks of data.

13 . The system of claim 8 , wherein the processing device is further configured to:

determine, for a plurality of directories in the dataset, a number of portions of the dataset having a size less than the threshold size; and

generate a plurality of data streams to transfer the dataset, wherein the portions of the dataset identified by the operation are distributed among the plurality of data streams based on the number of portions of the dataset in each directory having a size less than the threshold size.

14 . The system of claim 13 , wherein a number of portions of the dataset transmitted via each of the plurality of data streams comprises a substantially equal number of files.

15 . A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device to:

determine to transfer a dataset stored at a first storage system to a second storage system;

invoke an operation to retrieve, for a plurality of portions of the dataset having a size less than a threshold size, an identifier, metadata, and data stored in the plurality of portions of the dataset; and

transmit the identifiers, the metadata, and the data for the plurality of portions of the dataset having a size less than the threshold size to the second storage system.

16 . The non-transitory computer readable storage medium of claim 15 , wherein the operation comprises a single remote procedure call to the first storage system.

17 . The non-transitory computer readable storage medium of claim 15 , wherein the processing device is further configured to:

iteratively invoke the operation for one or more directories in the dataset at the first storage system.

18 . The non-transitory computer readable storage medium of claim 15 , wherein the processing device is further configured to:

concurrently invoke the operation on directories of the dataset to transfer the dataset over a plurality of communication channels in parallel.

19 . The non-transitory computer readable storage medium of claim 15 , wherein the processing device is further configured to:

determine, for a plurality of directories in the dataset, a number of portions of the dataset having a size less than the threshold size; and

generate a plurality of data streams to transfer the dataset, wherein the portions of the dataset identified by the operation are distributed among the plurality of data streams based on the number of portions of the dataset in each directory having a size less than the threshold size.

20 . The non-transitory computer readable storage medium of claim 19 , wherein a number of portions of the dataset transmitted via each of the plurality of data streams comprises a substantially equal number of files.