Communicating large amounts of data over a network with improved efficiency
View Patent ↗Disclosed herein are systems, methods, and computer-readable storage media for communicating large amounts of data. A system can first receive a request for data and determines if the data requested is above a threshold size. If the data is above the threshold size, the system can then prepare a message containing header information following a standard file transfer protocol. The system can also prepare metadata indicating a storage location and a size of the data requested. The message and the metadata can then be separately communicated to the requester. Upon receiving the metadata, a data stream can initiate the transfer of the data from the storage location to the requester, where the data stream removes unnecessary or redundant header information during the transfer from the storage location.
1. A computer-implemented method comprising:
receiving, by a processor, a request for specified data from a requester, wherein the specified data comprises multiple files;
determining, by the processor, that the specified data has a data size above a predefined first threshold and that each file in the multiple files is above a second threshold data size;
in response to determining that the requested specified data has a data size above the predefined first threshold:
preparing, by the processor, a message comprising header information associated with a file transfer protocol;
preparing, by the processor, metadata indicating at least a storage location of the requested specified data;
communicating the message and the metadata separately to the requester without the requested specified data; and
upon receiving the metadata by the requester, initiating transmission of the requested specified data from the storage location to the requester as a data stream, wherein the header information is omitted from the transmitted data stream.
2. The method of claim 1 , wherein the first threshold comprises at least a terabyte of data.
3. The method of claim 1 , wherein the requested specified data resides on shared content item management system.
4. The method of claim 1 , where the second threshold data size is 100 gigabytes of data.
5. The method of claim 1 , wherein the metadata further comprises a format of the data.
6. The method of claim 1 , further comprising identifying at least one of the location of the data and the format of the data prior to preparing the message.
7. A non-transitory computer-readable storage medium having stored therein processor-executable instructions for causing a processor to:
receive a request for specified data from a requester, wherein the specified data comprises multiple files;
determine that the specified data has a data size above a predefined first threshold and that each file in the multiple files is above a second threshold data size;
in response to the determination that the requested specified data has a data size above the predefined first threshold:
prepare a message comprising header information associated with a file transfer protocol;
prepare metadata indicating at least a storage location of the requested specified data;
communicate the message and the metadata separately to the requester without the requested specified data; and
upon the requester receiving the metadata, initiate transmission of the requested specified data from the storage location to the requester as a data stream, wherein the header information is omitted from the transmitted data stream.
8. The non-transitory computer-readable storage medium of claim 7 , wherein the first threshold comprises at least a terabyte of data.
9. The non-transitory computer-readable storage medium of claim 7 , wherein the requested specified data resides on shared content item management system.
10. The non-transitory computer-readable storage medium of claim 7 , where the second threshold data size is 100 gigabytes of data.
11. The non-transitory computer-readable storage medium of claim 7 , wherein the metadata further comprises a format of the data.
12. The non-transitory computer-readable storage medium of claim 7 , the computer-readable storage medium having additional instructions stored for causing a processor to identify at least one of the location of the data and the format of the data prior to preparing the message.
13. A system comprising:
a processor; and
a computer-readable storage medium having stored therein processor-executable instructions for causing a processor to:
receive a request for specified data from a requester, wherein the specified data comprises multiple files;
determine that the specified data has a data size above a predefined first threshold and that each file in the multiple files is above a second threshold data size;
in response to determining that the requested specified data has a data size above the predefined first threshold:
prepare a message comprising header information associated with a file transfer protocol; and
prepare metadata indicating at least a storage location of the requested data;
communicate the message and the metadata separately to the requester without the requested specified data; and
upon receiving the metadata by the requester, initiate transmission of the requested specified data from the storage location to the requester as a data stream, wherein the header information is omitted from the transmitted data stream.
14. The system of claim 13 , wherein the first threshold comprises at least a terabyte of data.
15. The system of claim 13 , wherein the requested specified data resides on shared content item management system.
16. The system of claim 13 , where the second threshold data size is 100 gigabytes of data.
17. The system of claim 13 , wherein the metadata further comprises a format of the data.